The outliers: do scientific trials obscure some good treatment results?
“It worked for me.” Almost everyone, sooner or later, feels like they got some blessed relief from a remedy of dubious value — something that’s not supposed to work according to the science, something “alternative,” maybe something straight out of the folk medicine of people from the other side of the world. Perhaps it was gua sha, or Epsom salts or a homeopathic arnica cream. Maybe it was something a bit more modern-seeming but damned with faint praise by the studies, like laser or shockwave therapy.
How could such a thing happen? Could the science just be wrong?
Could the studies have missed people like you?
It’s conceivable.
There are, of course, many ways for treatments to seem effective when something else actually deserves the credit. But most people don’t know about most of those, and even card-carrying skeptics often feel certain that the treatment worked, somehow, and the science be damned!
The #1 pretentious rationalization for these apparent successes is this classic half-truth about the limitations of science:
“But scientific trials show us the average response to a treatment. It may still work well for some people! It worked for me, after all.”
Or “it works for some of my patients” — because this idea is more often floated by professionals rationalizing their clinical choices.
This is the outlier gambit. It is often (suspiciously) extended into a rant against evidence-based medicine itself, scoffing at the futility of testing treatments at all. It’s not entirely wrong, but most people try to put much more weight on it than it can support.
The idea of “strong responders” is not crazy
The results of trials can indeed fail to reveal significant benefits for a few people who actually do well with a treatment that is unimpressive for most others. And I should probably acknowledge this more often than I do. Trial conclusions mostly focus on the typical response to a treatment, and, yes, sometimes the distribution of results shows high variance, and that spread does include a few genuine “strong responders” in the statistical Good Place.
The clearest cases of important “subgroups” of people who respond much better than average are pharmacological, where known biology makes their existence more plausible (if not almost a certainty). One of the all-time best examples comes from Yarnitsky et al, who showed in 2012 that patients with measurably poor endogenous pain inhibition — a testable glitch, not a guess — responded better to pain meds than patients without that problem. The response varied enormously depending on their capacity for inhibition, a critical clinical reality that was just invisible in the average effect.
That subgroup is real, the predictor is measurable, and the mechanism makes sense. (And no, unfortunately we don’t know why people have poor inhibition.)
Examples like that are a potent seed of truth for the outlier gambit.
Outliers may exist, but there are a lot of very big “buts” here (“and I cannot lie”)
Variance cuts both ways. Variance itself varies, and sometime the range of responses to a treatment isn’t always wide, and sometimes it just doesn’t reach far enough from the mediocre mean to include any strong responders — only “meh” or “slightly better than nothing” responders. It can also hide strong negative responders.
Many, if not most, outliers in small trials are noise, not signal. That noise is already routinely mined for “positive” results that are actually an illusion. They are routinely better explained by regression to the mean, natural recovery, reporting bias, co-interventions (and many other possible confounders) rather than a lucky treatment effect for a special few. Remember that trials routinely fail to effectively control for exactly the kinds of things that will produce meaningless outliers.
And then an independent team tested it (Hancock et al) and … sad trombone. No replication. The hope of an important subgroup, a signal strong enough for clear replication, evaporated on contact with fresh data. Bummer. (No really: it is a bummer. I’m not happy about this! I sincerely wish it had worked. Just like I truly wish there was a dinosaur hiding in a Scottish loch, because that would be really cool.)
There are quite a few other notable examples of failing to clearly identify clinically significant subgroups, which is why Rabey et al wrote, “while unidimensional subgrouping has been thought useful to guide treatment, it is unlikely to capture the full complexity of chronic low back pain.” It’s why Saragiotto et al concluded the quest for outliers is “far from optimal and not yet ready to be implemented in clinical practice.”
And so the widespread enthusiasm for special cases is a kind of special pleading. [Wikipedia] It’s what people propose in the conspicuous absence of validated subgroups.
You probably aren’t an outlier
Or, if you’re a clinician, your patients probably aren’t outliers.
If trial evidence is negative — or merely damns a treatment with faint praise, failing to impress — it's probably genuinely ineffective for most people, most of the time. The exceptions do exist, and maybe enough of them to justify cautiously trying iffy treatments once in a while, as long as they're relatively safe and cheap. But the exceptions mostly aren’t important exceptions.
Robust treatment benefits rarely lurk in the statistics, and bad clinical trial news is mostly as bad as it seems. The outlier hope is no more of a genuine win for healthcare than lotteries are a solution to poverty: yes, some people do win, but not many, not much, and not you.
Further reading
- The Power of Barking: Correlation, causation, and how we decide what treatments work — A silly metaphor for a serious point about the confounding power of coincidental and inevitable healing, and why we struggle to interpret our own recovery experiences
- Extraordinary Health Claims — A guide to critical thinking, skepticism, and smart Internet reading about health care
- Studying the Pain Studies — Tips and musings about how to understand (and write about) the extremely flawed science of pain and musculoskeletal medicine