[SciShow intro plays]

Hank: In 2012, a researcher named Glenn Begley published a commentary in the journal Nature. He said that during his decade as the head of cancer research for Amgen -- an American pharmaceutical company -- he’d tried to reproduce the results of 53 so-called “landmark” cancer studies. But his team wasn’t able to replicate 47 out of those 53 studies. That means that 88 percent of these really important cancer studies couldn’t be reproduced.

Then, in August 2015, a psychologist named Brian Nosek published another paper, this time in the journal Science. Over the course of the previous three years, he said, he’d organized repeats of 100 psychological studies. 97 of the original experiments had reported statistically significant results -- that is, results that were most likely caused by variables in the experiment, and not some coincidence.

But when his team tried to reproduce those results, they only got 36 significant results -- barely a third of what the original studies had found. It seems like every few months, there’s some kind of news about problems with the scientific publishing industry. A lot of people -- both scientists and science enthusiasts -- are concerned. So why does this keep happening? And what can be done to fix the system?

Right now, the scientific publishing industry is going through what’s being called a Replication, or Reproducibility Crisis. Researchers are repeating earlier studies, trying to reproduce the experiments as closely as possible. One group might publish findings that look promising, and other groups might use those results to develop their own experiments.

But if the original study was wrong, that’s a whole lot of time and money right down the drain. In theory, these repeat studies should be finding the same results as the original experiments. If a cancer drug worked in one study, and a separate group repeats that study under the same conditions, the cancer drug should still work.

But that’s not what happened with Begley’s cancer studies. And researchers in other fields have been having the same problem. They’re repeating earlier studies, and they aren’t getting the same results.

So why are these inaccurate results getting published in the first place? Well, sometimes people really are just making things up, but that’s relatively rare. Usually, it has to do with misleading research tools, the way a study is designed, or the way data are interpreted. Take an example from the world of biomedicine.

Researchers who work with proteins will often use antibodies to help them with their research. You might know antibodies as a part of your immune system that helps target outside invaders, but in scientific research, antibodies can be used to target specific proteins. And lately, there’s been a lot of evidence that these antibodies aren’t as reliable as scientists have been led to believe.

Companies produce these antibodies so researchers can buy them, and they’ll say in their catalog which antibodies go with which proteins. The problem is, those labels aren’t always right. And if researchers don’t check to make sure that their antibody works the way it should, they can misinterpret their results.

One analysis, published in 2011 in a journal called Nature Structural & Molecular Biology, tested 246 of these antibodies, each of which was said to only bind with one particular protein. But it turned out that about a quarter of them actually targeted more than one protein. And four of those... actually targeted the wrong kind of protein.

Researchers were using this stuff to detect proteins in their experiments, but the antibodies could have been binding with a completely different materials -- creating false positives and therefore, flawed results. That’s exactly what happened to researchers at Mount Sinai Hospital in Toronto. They wasted two years and half a million dollars using an antibody to look for a specific protein that they thought might be connected to pancreatic cancer.

Then they figured out that the whole time, the antibody had actually been binding to a different cancer protein, and didn’t even target the protein they were looking for. So the antibody-production industry is having some quality control problems, and it’s affecting a lot of biomedical research. Some companies have already taken steps to try and ensure quality -- one reviewed its entire catalogue in 2014 and cut about a third of the antibodies it had been offering.

Now, researchers could try testing the antibodies themselves, to make sure they only bind to the protein they’re supposed to. But that’s like conducting a whole separate study before they even get to start on the main project. Most research groups don’t have the time or money to do that.

But now that scientists are aware of the issue, they can at least be more careful about where they get their antibodies. Having accurate tools for research isn’t enough, though. Part of the reproducibility crisis also has to do with how experiments are designed.

This can be a problem in all kinds of different fields, but it’s especially an issue for psychology, where results often depend on human experience, which can be very subjective. Experiments are supposed to be designed to control for as many external factors as possible, so that you can tell if your experiment is actually what’s leading to the effect. But in psychology, you can’t really control for all possible external factors.

Too many of them just have to do with the fact that humans are human. One classic experiment, for example, showed that when when people had been exposed to words related to aging, they walked more slowly. Another research group tried to replicate that study and failed, but that doesn’t necessarily prove or disprove the effect.

It’s possible that the replication study exposed the subjects to too many aging-related words, which might have ruined the subconscious effect. Factors that weren’t directly related to the study could have also affected the results -- like what color the room was, or the day of the week. When such tiny differences can change the results of a study, it’s not too surprising that when they reviewed those 100 psychology papers, Nosek’s research group was only able to replicate 36 out of 97 successful studies.

But it also makes the results of the original studies pretty weak. At the very least, being able to replicate a study can show the strength of its results, so some scientists have been calling for more replication to be done -- in lots of fields, but especially in psychology. Because it can just be hard to determine the strength of the results of psychological experiments.

The fact that journals are so selective about what they publish is another reason the results of a study might turn out to be false. A lot of the time, researchers are pressured to make their findings look as strong as possible. When you publish papers, you get funding to do more research, and the more grant money you bring in, the more likely it is that the academic institution sponsoring you will want to keep you.

The problem is, journals are MUCH more likely to publish positive results than negative ones. Say you’re a biologist, and you spend three months working on a potential cancer drug. If after three months, you get positive results -- suggesting the drug worked -- then a journal will probably want to publish those results.

But if you end up with negative results -- the drug didn’t work -- that’s just not as interesting or exciting. So negative results almost always go unpublished -- which means that there’s a lot of pressure on researchers to conduct experiments that do have positive results. For example, Begley -- the biologist who led the cancer replication studies at Amgen -- tried an experiment 50 times and hadn’t been able to reproduce the original results.

The lead researcher on the original study told Begley that his team had tried the experiment 6 times, and gotten positive results once. So that’s what they published -- that one positive result. There’s so much pressure to publish significant findings that researchers might not always include all of their data in their analysis.

To get around this problem, some experts have suggested creating a new standard, where researchers include a line in their papers saying that they’ve reported any excluded data and all aspects of their analysis. If including that statement became standard practice, then if a paper didn’t have that line, that would be a red flag. But even if researchers do include all their data, they might just be doing the analysis wrong.

Because: data analysis involves math. Sometimes a lot of math. But researchers in a lot of fields -- like psychology and biology -- aren’t necessarily trained in all that math. You don’t always need to take courses in advanced statistical methods to get a degree in biology. So, sometimes, the data analysis that researchers do is just... wrong. And the peer reviewers don’t always catch it, because they haven’t been trained in those methods, either.

Then... there are p-values. The term p-value is short for probability value, and it’s often used as a kind of shorthand for the significance of scientific findings. To calculate a p-value, you first assume the opposite of what you want to prove.

So if you were testing a cancer drug, for instance, and you found that it kills cancer cells. To calculate the p-value for your study, you’d start by assuming that the drug doesn’t kill cancer cells. Then, you’d calculate the odds that the cancer cells would die anyway.

That would be your p-value. In other words, a p-value tells you the probability that the results of an experiment were a total coincidence. So for that cancer drug you’re testing, a p-value of less than .01 would mean that there’s a less than 1% chance the cancer cells would die even if the drug didn’t kill cancer cells. Usually, the standard for whether results are worth publishing is a p-value of less than .05 -- which would translate to less than a 5% chance of the cancer cells dying by coincidence. 5% is a 1 in 20 chance, which is pretty low, and there are lots of studies that get published with p-values just under .05.

Odds are that for at least a few of them, the results will be a coincidence -- and the findings will be wrong. That’s why a lot of people argue that p-values aren’t a good metric for whether results are significant. Instead, they suggest placing more emphasis on things like effect size, which tells you more than just whether an experiment produced some kind of change.

It tells you how big the change was. They also suggest more sharing of data -- including unpublished data -- something that’s gradually becoming more popular and accepted. So, yes -- there is a replication crisis, and it’s been highlighting a lot of problems with the scientific research and publication process. But scientists are also doing their best to solve it.

Thanks for watching this episode of SciShow, which was brought to you by our patrons on Patreon. If you want to help support this show, just go to patreon.com/scishow. And don’t forget to go to youtube.com/scishow and subscribe!