Thanks to Brilliant for supporting this episode of SciShow.

You can check out Brilliant.org/SciShow to learn more and get 20% off an annual premium subscription. This episode was filmed on March 19th, 2021.

For up-to-date information on the COVID-19 pandemic, check out our playlist linked in the description. [♪ INTRO]. If I offered you a choice between two things, and one of them was 95% effective and the other was 66% effective, you might wonder why I was even bothering. This is playing out right now with our vaccines for COVID-19.

For the sake of ease, we’re going to compare the three vaccines authorized for use here in the USA, but this basic logic applies to all of them, like Sputnik V or the vaccine made by AstraZeneca. The vaccine made by BioNTech and distributed here in the US by Pfizer has a 95% efficacy rate. And the one by Moderna clocks in at a similar-sounding 94%.

Meanwhile, the shot from Janssen, the pharmaceutical division of Johnson & Johnson, has a 66% efficacy rate, though their data is still awaiting peer review. But this is not a matter of you having a 5% chance of getting sick compared to a 34% chance. That’s just not how the numbers are calculated.

So let’s look at how it does work, why we can’t compare these numbers, and why that’s ok -- it’s still a good thing that we have these vaccines. This efficacy number comes from phase III clinical trials. Specifically, it’s from their primary endpoints, meaning the criteria they set ahead of time to determine a vaccine’s efficacy.

Phase three trials recruit thousands of volunteers, some who receive the vaccine and some a placebo. To calculate the overall efficacy of the vaccine, they take the number of cases of illness in the placebo group and calculate how much less likely the vaccinated participants were to get sick. So, let’s look at the BioNTech-Pfizer numbers as an example.

The vaccine and placebo groups had about 21,000 participants each. There were 8 positive COVID-19 cases in the vaccine group -- that’s not 5% of all participants! Meanwhile, 162 of those who didn’t get the vaccine got sick.

So 162 minus 8 equals 154 more unvaccinated people who got sick compared to the vaccine group. And if you divide that out, you get 95%. So it’s pretty simple, but the complications arise when you start comparing one trial to another.

Because each clinical trial arrived at that efficacy number slightly differently. Moderna and BioNTech looked for any symptomatic COVID-19 after two doses of vaccine or placebo shots, and counted those as cases in their trials. By contrast, the Janssen trial looked at efficacy against moderate to severe cases.

And they also had different criteria for when a case counted. Like, Moderna started counting cases that were contracted 14 days after participants got the second shot, while it was only 7 days for Pfizer. They had similar efficacy numbers, but under the hood, the counting doesn’t match up.

Not only that, but the trials all happened in different times, in different places, with different strains of the virus going around. This is why you can’t compare the numbers from one clinical trial to another, unless they were designed for it, which is hard to do when all the vaccines are all being invented at the same time. Talking about these numbers in the same breath is comparing apples to oranges, or durians to... something else.

BioNTech’s 95% is different from Janssen’s 66%... and different from Moderna’s 94%. So, if you’re thinking, “but 95% still sounds better,” well, there are a few more important points. All three of these vaccines were 100% efficacious at preventing death in the trials.

Less death and less burden on our hospitals are among the main reasons health officials are urging people to get whichever vaccine they’re offered. At least as long as they don’t have related allergy or other health concerns. And finally, there’s a difference between what happens in a controlled trial setting and the real world.

Think of efficacy as how well the vaccine works in trials, and effectiveness as how it works in the real world. We know one of those, but not the other -- yet. There are several variables that are likely to affect how things play out.

Like higher or lower proportions of vaccinated people in various communities, whether more contagious variants become dominant, and racial and other inequities in vaccination rates. Officials are keeping an eye on things. And in the meantime, I’ll be on the edge of my seat, and wearing a mask if that seat is in public.

But who knew you needed so much math to understand clinical trials? Well, it turns out if you want a more complex understanding of the world, you need an appreciation for the numbers that underlie it all. And Brilliant has everything you need.

From fundamentals to stats to complex numbers, they’ve got a course to help you learn all about it. If you want to check it out, you can go to Brilliant.org/SciShow. And if you sign up, you’ll get 20% off an annual Premium subscription. [♪ OUTRO].