Ioannidis: Making Medical Science Look Bad Since 2005

I am aware that John Ioannidis covered himself in shame during the COVID pandemic — not by being wrong initially, but by doubling down intensely, denying or justifying his errors, and becoming one of the most influential pandemic deniers and minimizers — a position that cost lives by undermining public health policies. See Dr. David Gorski's excellent analysis: What the heck happened to John Ioannidis? Fortunately, this article more or less holds up in spite of all this, but I at least want to acknowledge it.

PLOS Video Channel, CC BY 3.0, via Wikimedia Commons

Dr. John Ioannidis.

In 2005, PLoS Medicine published a now-famous paper with the attention-grabbing but exasperating title, “Why Most Published Research Findings Are False”.1 It was written by John Ioannidis, MD, PhD, an American doctor from Greece, a distinguished author of about 400 peer-reviewed papers, 40 books and book chapters or so, and much more. Ioannidis is a giant — albeit somewhat reduced in stature post-pandemic2 — and I am really nobody to criticize.

But, oooh, I hate the title of that paper.

This intensely intellectual article — it’s completely, hopelessly nerdy — became one of the most downloaded articles in the history of the Public Library of Science and was described by the Boston Globe as an instant cult classic.

I like the content of the paper very much, but the title is a misleading generalization that, unfortunately, plays right into the hands of quacks and charlatans, who like nothing better than to cite any expert who seems to be saying that science is so flawed — that “science is wrong” — that it can’t be used to debunk their nonsense. This is their sole interest in the paper, and indeed it has been extensively cited co-opted for this very purpose. A substantial portion of its popularity is probably due to how effectively its title can be used to undermine the credibility of science.

Ironically, while this work should serve to improve the quality of scientific medicine, it is being used by some cranks to attack the scientific basis of medicine.

Dr. Steven Novella, from an article quite similar to this one, Are Most Medical Studies Wrong?

I don’t believe that it’s an option for scientists to be oblivious to these issues — it’s naïve to think that a title like that won’t be abused, and it’s naïve to think it doesn’t matter.

It will be abused, and it does matter.

Science is not wrong

Attention: the paper does not say that “science is wrong”!

What Ioannidis really says is much less ominous: he argues that it should take rather a lot of good quality and convergent scientific evidence before we can be reasonably sure of a “scientific fact,” and he presents good (scientific!) evidence that a lot of so-called conclusions are premature, not as ready for prime time as we would hope.3 Any supposedly large treatment effect in medical research is probably exaggerated, and “when additional trials are performed, the effect sizes become typically much smaller,” so “well-validated large effects are uncommon.”4 Extremely uncommon, in fact.

It’s okay for science to work like this. Ioannidis did not mean that science is broken or deeply flawed.

In fact, this is actually how science should work, and it’s not the least bit surprising to good scientists, who never claimed in the first place that their results are infallible or that their conclusions are “true.” When most scientists publish, they’re just saying, “Hey, here’s what we did, here’s what we saw — what about you guys?” And then the research community responds, hopefully, with thoughtful criticism, and more research.

“Scientific investigation is the noblest pursuit,” Ioannidis writes.5 “I think we can improve the respect of the public for researchers by showing how difficult success is.”

Sadly, not all scientists are good at their jobs.

A medical Madoff

Credit for that excellent heading goes to Scientific American, which reported on “a medical Madoff,” a doctor who published several influential but entirely fraudulent scientific papers:

Over the past 12 years, anesthesiologist Scott Reuben revolutionized the way physicians provide pain relief to patients undergoing orthopedic surgery for everything from torn ligaments to worn-out hips. Now, the profession is in shambles after an investigation revealed that at least 21 of Reuben’s papers were pure fiction, and that the pain drugs he touted in them may have slowed postoperative healing.

A Medical Madoff: Anesthesiologist Faked Data in 21 Studies, Borrel (Scientific American)

The story of Scott Reuben’s fraud will also be co-opted to make the point that “mainstream” medical science is rotten. But it wasn’t “science” that was bad, it was Scott Reuben. The “science” wasn’t misleading, Scott Reuben was misleading. Scott Reuben was a liar-liar-pants-on-fire — and thank goodness science is self-correcting by nature, or who knows how much longer he would have gotten away with it.

Reuben’s bogus results were more or less doomed to be debunked. Science usually exposes scientific frauds quite quickly, and this is a much more important characteristic of science than the fact that frauds occur in the first place. Like a bad organ transplant, the scientific community will reject frauds the moment they are detected. This is strikingly different from the world of alternative medicine, where quacks can get away with selling non-medicine for decades, because you cannot discredit someone who had no credit to begin with — someone who never belonged to any respectable community or adhered to any standards in the first place, someone who may indeed define themselves in opposition to professional communities and standards. You can’t boot someone out of a club they never joined, and so they are “regulated” only by popular opinion and market forces — and our culture basically gives them a pass on proving anything, and if doctors and scientists criticize them it’s just a “conspiracy.”

This is why prosecutions of con-artists in health care are remarkably rare. And it’s why anything that undermines the credibility of science in public opinion is an aid and comfort to quacks.

Freedman’s Atlantic article about Ioannidis

In 2010, David Freedman authored an article about Ioannidis for The Atlantic, Lies, Damned Lies, and Medical Science, which got a great deal of attention at the time and continues to be cited widely. It is basically a translation of Ioannidis’ key concepts for a general readership. Unfortunately, it sensationalizes and emphasizes the “science is wrong” message … but on a much larger scale than ever before.

Freedman’s slant is subtle. He is not wearing an anti-scientific agenda on his sleeve here. (Update: he is now. In the July/August 2011 Freedman published in The Atlantic again, this time as a much more blatant apologist for alternative medicine.) He gets a lot of things exactly right, and concludes with lovely quotes from Ioannidis that go a long way to balancing the article. I don’t doubt that many readers actually got the right idea, and that is great.

But don’t underestimate the intensity or importance of the “war” between mainstream and alternative medicine — a much hotter war than most people are aware of — and the degree to which irrational, unethical quacks will make hay from any shred of an argument against science. Unfortunately, Freedman’s treatment of Ioannidis gives great ammunition to anyone who has a chip on their shoulder about mainstream medicine. Citing Freedman gives them a new way to miss Ioanndis’ real point with near impunity, and sound good doing it. I’ve already seen it cited extensively in that way.

Unfortunately, we also know from some of Freedman’s other publications that he has does indeed have a clear anti-scientific bias. For example, he tries to make the case that Wakefield’s dangerous vaccine-autism link fraud is a meaningfully representative example of a general problem with medical science. You really have to be banging an anti-science drum hard to get that out the extreme Wakefield example! Every statement Freedman makes in that post has truth in it, but he’s bending over backwards to make science look as untrustworthy and corrupt as he possibly can — as if two out of every three scientists is a Wakefield!

Once you know this about Freedman, it’s hard not to see how he used his Atlantic article as an opportunity to sensationalize the wrongness of science. And the proof is in the way the quacks have been gleefully citing it every chance they get. One quack will post it on Facebook, say, and get a long string of "amens" about the evils of science and "mainstream" medicine.

That’s a very unfortunate outcome ... especially considering where Ioannidis is really coming from.

Human overconfidence and fallibility

Fortunately, outright scientific fraud is actually quite rare — and it was not the subject or point of Ioannidis’ paper. His point was that a lot of science is done amateurishly to “prove” pet theories, and this is tends to produce incorrect results, surprise surprise. Dr. Gorski phrased it a more precisely:

What Ioannidis’ research shows is that clinical trials examining highly improbable hypotheses are far more likely to produce false positive results than clinical trials examining hypotheses with a stronger basis in science.

Science is a human enterprise, and people are imperfect (you may have noticed). Honest, well-intentioned, competent scientists can have strong egos and biases, and even the best can make logical mistakes and become locked into their prejudices.6 Many good scientists play into the hands of critics with overconfident “conclusions” that are all-too easy to point to as examples of scientific inconsistency and imperfection.

And that’s the good scientists.

Overconfidence and premature conclusions are what Ioannidis is correctly warning against. This is why I have criticized so many scientific papers. Here’s a good example, with the cockiness built right into the title: “Radial extracorporeal shock wave therapy (ESWT) is safe and effective in the treatment of chronic recalcitrant plantar fasciitis.”7 The evidence in that paper was nowhere near enough to justify that title. It wasn’t fraudulent, just overconfident.

So it’s ironic that Ioannidis’ title was also brash and irresponsible.

Piling irony on irony, it’s possible Ioannidis may not have even written his own title — an PLoS Medicine editor may have done it. Science writer and broadcaster Flora Graham points out she doesn’t write her own headlines:

The subeditor does that! Writing a good headline is much harder than it looks. I have felt that my carefully measured story has been given an inflammatory or inaccurate headline — and given readers the wrong idea. What if Ioannidis’ editor wrote his headline? His paper cautioning against over-interpretation of evidence ends up with a title that over-interprets his evidence!

Ooooh, that rotten “media”!

Usually it is everybody’s favourite scapegoat, the media — not scientists themselves — who unwisely portray research findings as “conclusions.”

Sure, sure, almost every summary (abstract) of a scientific paper has a “conclusion.” But good scientists know full well that all knowledge is tentative, provisional. They know that even the best evidence is just a piece of an evolving (and fascinating) puzzle. But reporters want to make it look like it’s all a done deal! This exacerbates the public perception that science is unreliable and inconsistent and that scientists, like economists, can never agree on anything, the never-ending battle of the experts. And this is good, rich soil for evangelicals and quacks to sow doubt and dissent about evolution and scientific medicine. The average person is much more prone to getting suckered into buying quackery when scientific medicine is constantly portrayed as chasing its own tail.

So it really troubles me when scientists themselves, even incredibly smart guys like Ioannidis — or his editor? — hand the enemy an intellectual bazooka on a silver platter.

About Paul Ingraham

I am a science writer in Vancouver, Canada. I was a Registered Massage Therapist for a decade and the assistant editor of ScienceBasedMedicine.org for several years. I’ve had many injuries as a runner and ultimate player, and I’ve been a chronic pain patient myself since 2015. Full bio. See you on Facebook or Twitter., or subscribe:

SUBSCRIBE

Get PainSci updates in your inbox, mostly small, 0–3/week depending on coffee dosage. More info.

Related Reading

• Statistical Significance Abuse — A lot of research makes scientific evidence seem much more “significant” than it is. An essential companion to this article.
• Why “Science”-Based Instead of “Evidence”-Based? — The rationale for making medicine based more on science and not just evidence… which is kinda weird.
• “Evidence-based medicine has been hijacked: a report to David Sackett,” Ioannidis, J Clin Epidemiol, 2016.
• Most Pain Treatments Damned With Faint Praise — Most controversial and alternative therapies are fighting over scraps of “positive” scientific evidence that damn them with the faint praise of small effect sizes that cannot impress
• Alternative Medicine’s Choice — What should alternative medicine be the alternative to? The alternative to cold and impersonal medicine? Or the alternative to science and reason?
• Science versus Experience in Musculoskeletal Medicine — The conflict between science and clinical experience and pragmatism in the management of aches, pains, and injuries

Notes

1. Ioannidis J. Why Most Published Research Findings Are False. PLoS Medicine. 2005 08;2(8):e124. PainSci Bibliography 55463 ❐

2. Dr. John Ioannidis (born 1965 in New York City) is a professor and chairman at the Department of Hygiene and Epidemiology, University of Ioannina School of Medicine as well as tenured adjunct professor at Tufts University School of Medicine and Professor of Medicine and Director of the Stanford Prevention Research Center at Stanford University School of Medicine.

   In the COVID pandemic, Ioannidis alienated many who had previously respected him. This began reasonably enough when he expressed concern early on about public health policy racing ahead of the evidence, but evolved into something more stereotypically and ironically anti-scientific, until he had strange bedfellows. Dr. David Gorski for ScienceBasedMedicine.org:

   John Ioannidis is one of the most published and influential scientists in the world, someone whose skewering of bad medical research we at SBM have frequently lauded over the years. Then the COVID-19 pandemic hit. Since then, Prof. Ioannidis has been publishing dubious studies that minimize the dangers of the coronavirus, shown up in the media to decry “lockdowns,” and, most recently, “punched down”, attacking a graduate student for having criticized him. What happened? Did Prof. Ioannidis change, or was he always like this and I just didn’t see it? Either way, he’s a cautionary tale of how even science watchdogs can fall prey to hubris.

3. But even Ioannidis’ research findings about the wrongness of research findings may be wrong! And precisely this point has been raised by some critics. In 2007, Goodman and Greenlander wrote: “we agree with the paper’s conclusions and recommendations … but calculating the unreliability of the medical research literature, in whole or in part, requires more empirical evidence and different inferential models than were used. The claim that ‘most research findings are false for most research designs and for most fields’ must be considered as yet unproven.” Ioannidis replied substantively, and the debate goes on … as it should.
4. Pereira TV, Horwitz RI, Ioannidis JPA. Empirical evaluation of very large treatment effects of medical interventions. JAMA. 2012 Oct;308(16):1676–84. PubMed 23093165 ❐
5. Ioannidis JPA. Why most published research findings are false: author's reply to Goodman and Greenland. PLoS Med. 2007 Jun;4(6):e215. PubMed 17593900 ❐ PainSci Bibliography 55643 ❐
6. None more famously than Albert Einstein, who produced more good ideas in his 20s than most universities can produce in a decade, but still couldn’t accept the implication of quantum mechanics that “God” does indeed “play dice”.
7. Gerdesmeyer L, Frey C, Vester J, et al. Radial extracorporeal shock wave therapy is safe and effective in the treatment of chronic recalcitrant plantar fasciitis: results of a confirmatory randomized placebo-controlled multicenter study. Am J Sports Med. 2008 Nov;36(11):2100–2109. PubMed 18832341 ❐ PainSci Bibliography 56185 ❐

   This overconfidently titled paper essentially declares that there is no longer any controversy about radial extracorporeal shock wave therapy for plantar fasciitis (note that this is not quite the same thing as “regular” ESWT, but they are close cousins — basically the same idea, just different settings).

   My confidence in their conclusions is suppressed by the fact that the researchers are on the payroll of a company that makes these devices, and the entire study was funded by that company. As always, conflicts of interest are not necessarily a deal-breaker, but they can be, and this one seems particularly strong.