Last week, we discussed systematic reviews, and why they’re better than review articles, or opinions. But they’re not the only types of “studies of studies” we’ve looked at. Sometimes you can go a step further. After you’ve collected all the appropriate research, you can merge the data together and do one large analysis. Those studies are called meta-analyses, and they’re the topic of this week’s Healthcare Triage.

(Theme music / montage)

So you've done your systematic review, you've gathered all the studies together, some show one thing, some show another, but what's truth?

Well, with a meta analysis, you put all the data into one pool.  This allows for many smaller studies which might not have much power, to be merged into a large study that allows for more comprehensive analyses.  It also allows for research that on its own might not be robust to achieve some sort of statistical significance.

You get a diagram like this: 

This is a study that looked at whether vitamin D supplementation affected bone density.  This is called a pools odd ratio diagram.  The chart shows you the odds ratio (highlighted) for each outcome - one study per line.  If the square and error bars for a study are entirely to the right of the vertical line, then vitamin D was beneficial.  

If the square and its error bars are entirely to the left of the line, then vitamin D was detrimental.  If the square, or its error bars, in any way, touch the line then it's a non-significant result.

At the bottom, the results are summed into one large diamond with the same rules applying.  As you can see there was a small benefit, weighted mean difference 0.8%, for studies looking at the effect of vitamin D supplementation on bone mineral density of the femoral neck.  

Woo-hoo (sarcastic).

Hip? No effect.

Trochanter? No effect.

Lumbar spine? No effect.

Forearm? No effect.

We covered this in a previous episode about how milk is sort of ridiculous.  Deal with it!

But that doesn't mean that systematic reviews are infallible or immune from criticism.  In fact, often the results of a systematic review or meta-analysis can be hotly contested.  

As an example, I'd like to revisit organic food.  As we discussed on the episode of Healthcare Triage about organic food, in 2009 a group of scientists published a major review of organic versus conventionally grown food, covering research from 1958 through 2008.  

They reviewed 52,471 articles and found 162 studies that compared crops and livestock products. They deemed 52 of them high enough quality for inclusion in their analyses. They found no significant difference between organic and conventionally grown food with respect to nutrient content.

This study was considered by some to be methodologically imperfect. Therefore, researchers from Stanford University published another systematic review and meta-analysis in 2012.  They reviewed research through May of 2011, found 460 studies, and identified 237 that met their inclusion criteria. They found a lack of evidence that organic foods are significantly more nutritious than conventionally grown food.

But very recently a new study in the British Journal of Nutrition declared that research said that organic fruits and vegetables are more nutritious. Press releases declared it the largest study of its kind. Because of its size, and its breadth, some declared that it trumped previous research that showed that organic food didn't appear to be any safer or more nutritious than conventionally grown food.

Some of you even claimed that our episode on the topic was wrong.

The authors of this very new paper acknowledge that the previous views existed but claimed that they weren't comprehensive enough.  

They searched the literature from 1992 through 2011 and reviewed 448 studies that they found. They deemed 343 of them appropriate for inclusion, which did make this a "larger study".  

But this study didn't include much more newer data than the Stanford study did. It simply had more data because it was more permissive of the type of studies that it deemed of high enough quality to be included.  

Their analysis found that there were significantly higher levels of anti-oxidants in organic food than in conventionally grown food. It's on this basis that they declared organic food was more nutritious. They also found higher levels of pesticides on conventionally grown food which they said made them more unsafe.

It's important to be realistic about what anti-oxidants can and can't do. They're a type of compound, used by our body, to fight against "free radicals," or chemicals that can cause damage to many structures by stealing electrons from certain molecules. Antioxidants can give electrons to free radicals, so they don't take them from our bodies. But antioxidants aren't nutrients. They also aren't all the same. Each one works in a certain way in different parts of the body. More importantly, there's very little evidence that supplementation with them leads to better health.

Vitamin E has shown mixed results in the Women's Health Study, the Heart Outcomes Prevention Evaluation trial, and the GISSI-Prevenzione trial. Beta-carotene was shown to have no effect on heart disease or cancer. Mixtures of antioxidants didn't prevent cardiovascular events in women or cancer, heart disease, or death in anyone.

These studies all included much larger doses of antioxidants than would likely be received by eating organic fruits and vegetables as well.

Second, this much more recent meta-analysis also found that organic crops are lower in protein. That's an actual nutrient, and it's being ignored in much of the reporting on this study.

Third, while levels of pesticide may be higher in conventionally grown food, none of the studies have detected levels of chemicals that approach anything near what we would be classified as an "unsafe level".

Finally, though, this study provides an opportunity to understand how we might think about systematic reviews and meta-analyses in general. If it were patently obvious that organic foods were nutritionally superior, we would need no meta-analysis. Large studies would find clear benefits with respect to nutrients, and that would be that. We're having this argument because it is hard to find a benefit.

Moreover, when a new systematic review or meta-analysis finds a benefit that an old study didn't by being more permissive of the research it includes, that should give us pause. It's entirely possible, of course, for previous work to be flawed, and to have left out critical research. But that doesn't appear to be the case here. The new study included everything the old study did, and then added to it research that didn't make the cut the first time. That's potentially problematic.

Of course, it's a judgment call as to which analysis is correct. I tend to favor the Stanford study because it seems like it was more rigorous in excluding studies with weaker methodologies.

This is one of those times, however, when people's predisposed beliefs will likely color their interpretations of which study is correct. Even if you favor the newer one, the differences, while statistically significant, have no evidence to support their leading to any real health benefits.

I would be remiss if I neglected to mention one more thing. The Stanford study was done with no extra funding at all. The newer study, though, cost $429,000., and was funded by the charity that supports organic farming research. That doesn't mean that a conflict of interest tainted the methods or results, but it should at least be acknowledged.

[outro]