Quick Hit Summary

Although it's not the most exciting topic, the ability to understand research is critical to understanding the claims made by popular newspapers/magazines, supplement manufactures, etc. There are two main types of scientific studies: epidemiological and experimental. Epidemiological studies look at ASSOCIATIONS between 2 variables; they do not show a cause and effect relationship. An example would be a study showing that stronger individuals eat more protein in their diet. In contrast, experimental studies show CAUSE AND EFFECT relationships. An example of this study design would be that ingesting 30 grams of protein increases muscle protein synthesis in 20 study participants. Other key issues to keep in mind when evaluating a study include… Did the study include a large number of participants or only 5-10? Are the characteristics of the participants (age, gender, possibility of a disease such as type II diabetes etc) similar to your own? If the answers to these questions are "yes", than the results are more likely to apply to your life.

The Importance of Research

Figure 1 Scientific Studies Determine That Only Science Gives Legitimacy²

I realize that research terminology isn’t exactly the most exciting topic of conversation. In fact most people probably find it downright BORING. While completing undergraduate studies, I had numerous research based lectures that broke down the nitty-gritty nuances of one study design vs. another. These were definitely the days where my pre-lecture preparation required a double shot of espresso if I wanted to stand any chance staying awake! As tough as those lectures were to stay awake for, I’ve found them vital to my ability to properly interpret information that I read both in scientific journals as well as magazines (Muscle & Fitness, Men’s Health, etc).

I draw a parallel between this article and the movie Gladiator. I fondly recall my thoughts from watching this movie for the first time. As the movie started, I was completely blown away while the Romans went into battle with the Barbarians. During the next 45 minutes, the film struggled to keep my interest as the plot unfolded between Maximus (Russell Crowe) and the new emperor Commodus (Joaquin Phoenix). An hour into the film, things start to heat up again and ~ 3 hrs after the film started, I had a new favorite movie. However, without the 45 minutes of “mind numbing” plot development slipped in there, this would have just been another action film, without any distinction from all the previous ones I’ve seen.

Now back to research terminology… Similar to the middle part of Gladiator, it’s not the most exiting topic. However, just as this part was crucial for appreciating the depth of the Maximus’s plight, understanding research terminology is critical for recognizing the true value of a given study’s results. Although my intention is not for you to memorize all the information presented here, I hope that you'll bookmark it and use for reference purposes.

So without further adieu I bring you:

Research Study Design 101

I’m going to list the different types of studies that one may come across in descending order; starting with the weakest and finishing with the strongest study designs¹. There are 2 broad types of study designs, epidemiological and experimental; both of which can be broken down into further subcategories. To make things a little easier, give the same basic example for all of the studies, but tweak it a little to fit the study design. Also, I’ll give strengths and weaknesses of each study type.

STUDY DESIGN TYPE 1: EPIDEMIOLOGY STUDIES

Epidemiology studies: Study design that looks at ASSOCIATIONS between 2 variables. It CANNOT PROVE CAUSE AND EFFECT.

————————————————————————————————————————————————
1a. Case Studies: Anecdotal evidence based off one person’s experience.

Example: I lift heavier weights during my workouts when I drink an energy drink vs. plain water.

Strengths: Provides ideas for further research.

Weaknesses: Based off the results obtained by a single individual. Results cannot be generalized over to other individuals

————————————————————————————————————————————————
1b. Cross Sectional (prevalence) Study: a study looking at the relationship between a given population and a specific characteristic/variable. Only looks at one specific instance in time.

Example: At the USA Powerlifting championships each 1st place lifter is asked if they’re CURRENTLY taking a post workout protein shake.

Strengths: Quick and easy method to find association between populations and a given variable.

Weaknesses: No time relevance/directionality is present in study. Thus, can’t say what caused what; kind of like the chicken or the egg debate. One can only say that a relationship exists between 2 variables.

————————————————————————————————————————————————
1c. Case Control (retrospective study) Study: Upon completion of an event, researchers look at those who were and were not exposed to a given element prior to a certain event. Results obtained by each group are then compared.

Example: UPON THE COMPLETION of the USA Powerlifting Championships each lifter is asked if they took a post workout protein & carbohydrate shake during the three months leading up to the event. Researchers then examine if those who took the post workout protein shake were more likely to be champions than those who didn’t take a post workout shake.

Strengths: Stronger than cross sectional study because a time element is involved. Thus, one can determine the order that the events occurred (ie- using our example, one now knows that some individuals were taking shakes prior to the day of the event.)

Weaknesses: Several confounding variables are involved. In particular is recall bias which occurs when an individual inaccurately reports if they had been exposed to a variable prior to the event. This may be as simple as forgetting if they took a given supplement on a consistent basis (ie- did they take a general protein blend recovery drink or was it a whey protein). Inaccurate recall also occurs when the variable they were exposed to is frowned upon by the general public. For example weightlifting champions are asked if they took steroids prior to the Olympics. If the individual did take steroids, it’s unlikely that they’ll tell the truth for fear of public scorn.

————————————————————————————————————————————————
1d. Cohort (prospective) Study: the reverse of a case control study. Researchers record differences between individuals prior to a given event. This study design is considered stronger than a case control study.

Example: 3 MONTHS PRIOR to the USA Powerlifting Championships, researchers ask lifters if they plan to take daily post workout whey protein & carbohydrate shakes leading up to the championships. After the completion of the championships, they compare the results of those who took the shakes vs. those who did not take the shakes.

Strengths: eliminates recall bias. Provides time element.

Weaknesses: still does not prove direct evidence of cause & effect.

STUDY DESIGN TYPE 2: EXPERIMENTAL STUDIES

Experimental Studies: Study design that looks at how 1 variable directly influences another variable. Only type of study providing evidence of direct CAUSE AND EFFECT relationship.

————————————————————————————————————————————————
2a. Randomized Controlled Laboratory Study: Individuals are randomly assigned to either a control group (receives sham/placebo treatment) or treatment group (receives unique treatment that study is studying). However, rather than living in their normal environment, everyone lives in the same controlled environment to ensure that they are exposed to the same elements. Often, these studies are done on animals rather than humans.

Example: Researchers examine the effect post workout protein and carbohydrate supplementation has on muscle gain when taken over the course of 3 months. Rats receive the same exercise routine and take their respective supplement (placebo or protein powder) after running on a mousewheel. Over the course of the day, each rat is given to the same diet, exercise routine and sunlight exposure. Muscle dimensions are taken at the start and end of the study. After 5 weeks of following this protocol. researchers compare gains made by each group to see if protein & carbohydrate supplementation resulted in bigger muscle gains.

Strengths: Researchers are able to limit factors that may affect the studies outcome (such as diet, etc).

Weaknesses: Humans don’t live in a controlled environment, making it difficult to say with certainty that the results from these study’s can be applied to us.

————————————————————————————————————————————————
2b. Randomized Clinical Control Trials: This is considered the GOLD STANDARD with respect to study design. Individuals are randomly assigned to either a control group (receives sham/placebo treatment) or treatment group (receives unique treatment that study is studying). Each group then goes back to their normal every day routine, while completing the specified treatment assigned to them.

Example: Researchers are studying the effects that post workout protein & carbohydrate supplementation has on muscle gain when taken over the course of 3 months. Participants of similar physical fitness capabilities complete the same exercise routine, take supplements (placebo or protein powder) as part of their normal every day life. Muscle dimensions are taken at the start and end of the study. After 3 months of this routine, researchers compare gains made by each group to see if protein supplementation resulted in bigger muscle gains.

Strengths: Attempts to prove if research results obtained from a laboratory can be reproduced in a “real world” environment.

Weaknesses: Unable to rule out all factors that could confound study results. (eg- control group could show bigger muscle gains simply by eating an overall healthier diet that group receiving protein).

Other Issues #1: In-vitro vs. In-vivo tests

In vitro: Research conducted on NON-LIVING organisms. Studies are often conducted in test-tubes or petrie-dishes.

Example: Researchers measure the effects of lactic acid on muscle fatigue in muscle tissue obtained from a frog

Strengths: Direct results are easy to obtain. In vitro tests often serve as the foundation for later testing.

Weaknesses: It's a large leap to apply research obtained in vitro directly to living organisms.

———————————————————————————————————————————-
In-vivo: Research conducted on LIVING organisms.

Example: Researchers measure the rate of protein synthesis in muscle tissue 30 minutes after participants complete an intense lifting session

Strengths: Results obtained in living organisms can easily be applied

Weaknesses: No weaknesses present. Weaknesses only exist in how one applies a study's results.

Other issues #2: Validity

Validity: considered to be a measure of the amount of faith we can put in the findings of a study. In other words, how "true" are a study's results. In the end, the strength of the study is determined by its validity.The 2 types of validity that we are generally concerned with are internal and external validity
————————————————————————————————————————————————
2 types of validity:

1. Internal Validity: this type of validity examines how well a study is designed. For instance, were valid measurement tools used? Were enough participants used? Were participants of similar backgrounds? Were participants "blinded" with respect to which treatment they received (ie- supplement or placebo)?

Examples:

Good internal validity: Researchers study the effects of a post workout shake on body composition changes. Baseline levels of fat mass and fat free mass (ie- muscle, bones, etc) were obtained using DEXA, a tool clinically proven to determine body composition. 20 individuals of similar backgrounds (female, 20-25 years old, 3 years of lifting experience) were randomly assigned to receive either a placebo or the true supplement post workout. All participants completed the same workout. Additionally, drinks were consumed prior to leaving the training/testing facility, in front of supervisors to ensure compliance. After 3 months of training, researchers compared changes in body composition between the groups

Bad internal validity: Researchers study the effects of a post workout shake on body composition changes. Baseline levels of fat mass and fat free mass (ie- muscle, bones, etc) were obtained using skin calipers, a tool clinically proven to be less accurate than DEXA at assessing body composition. 20 individuals of differing backgrounds (female, 20-45 years old, 0-5 years of training experience) were assigned to receive either a placebo or the true supplement post workout. Participants completed a resistance training routine based off their preferences. Upon completing their workouts, each lady grabbed their shakes as they headed out the door. After 3 months of training, researchers compared changes in body composition between the groups

———————————————————————————————————————————-
2. External Validity: The measure of how well the results obtained from a study populations apply to the general population. Results from a study should only be applied to those who have characteristics similar to the population the research was conducted on. Due to physiological differences between men & women, trained vs. untrained individuals, etc, one cannot expect results obtained by one specific study population to be reproduced in another group of individuals.

Examples:

Good external validity: Overweight men, ages 25-30 years old, with no prior resistance training experience, take post workout protein and carbohydrate supplementation after workouts for 3 months. Final results show that those who took the protein and carbohydrate shakes have significant increases in muscle size vs. those who received the placebo. Researchers conclude that overweight men, between the ages 25-30, with no prior resistance training experience+ can expect gains in muscle size when taking post workout protein & carbohydrate shakes.

Bad external validity: Overweight Men, ages 25-30 years old, with no prior resistance training experience, take post workout protein and carbohydrate supplementation after workouts for 3 months. Final results show that those who took the protein and carbohydrate shakes have significant increases in muscle size vs. those who received the placebo. Researchers conclude individuals can expect increases in muscle size when consuming post workout shakes …..***Notice how this study merely said "individuals" vs. naming a specific population. A middle aged female with 12% body fat and 4 years of training experience should not assume that she would obtain body composition changes similar to what was observed in the men included in the study.

One has to be VERY CAREFUL with external validity issues; especially when reading advertisements. Numerous times I've seen supplement companies promote a product with research conducted on one population (eg- inactive obese individuals) and then advertise the product to individuals with completely different backgrounds (eg- highly active, lean athletes). Similarly, certain supplements have staked their claim based solely off in-vitro studies!

Other issues #3: Sample Size

Sample Size: Sample size should actually fall under the "Validity" section. As one would naturally guess, sample size is simply the number of people that participate in the study. However, I feel many individuals jump to conclusions based off studies using questionable sample sizes. Thus, I want to make a quick note regarding sample size…

A couple of problems exist in studies with small numbers. The first issue ties in with external validity. Individuals are not genetic clones of each other (although identical twins are pretty dang close). Even if study participants have similar background (eg- males, 20-25 years of age, 3+ years of resistance training), major differences still exist between individuals. Thus, in larger sample sizes, more variability is present, increasing the likelihood that the results are applicable to you. For example, would you be more confident that a supplement was legit if it significantly improved muscle growth in a study involving 5 vs. 20 individuals?

The second problem with small sample sizes is that the results from 1 or 2 individuals can greatly skew results. For instance, lets say a study measuring the effects of supplement "X" on improved bench performance assigns 4 individuals to each respective study group (control & experimental). After 4 weeks of supplementation, 1RM are measured and compared to values obtained at the start of the study. Researchers find the the following changes in bench press strength (measured in pounds) at the conclusion of the study:

• Control Group (received placebo): 5, 5, 0, 0,

• Experimental Group (received supplement): 0, 5, 5, 30

After going through statistical test, its determined that taking supplement X significantly improved bench press strength. As we can see though, 1 individual in the experimental group had an extreme increase in bench press strength (30 lbs). If we exclude him/her from study results, we see that results are not significantly different between groups. With larger sample sizes, outliers still appear. However, as a sample size increases, the effects of a single extreme outcome tend to be minimized. Thus, it's less likely that statistically significant results will occur when no true difference really exists.

So what do I consider to be a small sample size? Unfortunately I cannot give you a set number. It varies from study depending on what's exactly being examined. However in a typical study involving, "The effects of supplement X on increased performance" I generally like to see AT LEAST 10-20+ participants per study group. I'm not saying studies containing <10 participants per study group are necessarily "bad" studies. I'm simply saying more potential validity issues are present.

Where Do We Find Studies?

Now that you have a little better of an idea on how to find studies, you're probably asking yourself, "Where do I find scientific studies?" or similar questions. The answer is pubmed.gov. It can be thought of as the "google" of scientific research. If you're not quite familiar with how to navigate around it, you may be interested in checking out my article, Pubmed for Dummies.

Bottom Line

Well, its been a long journey through research terminology. Hopefully, this article was able to shed a little light on different types of studies. As detailed above, every study design has its own strengths and weaknesses. Thus, do not form rock hard opinions based off a single study. Rather a collection of studies are needed before one can form solid conclusions about a given topic.

References

¹ Gay, John.Clinical Epidemiology & Evidence-Based Medicine Glossary:Clinical Study Design and Methods Terminology.Washington State University.Copyright 1998-2009. Obtained from "http://www.vetmed.wsu.edu/courses-jmgay/GlossClinStudy.htm"

² Accessed May 31, 2010 from: http://www.flickr.com/photos/scientificstudies/3388735309/

³ Thumbnail photo image- created by Andrew Baron. Accessed May 30, 2010 from:www.flickr.com/photos/andrewbaron/2083620607/