“Positive”

Most slightly positive study results are actually just bogus. The weaker a positive result to a scientific test of a treatment, the more likely it is to simply be wrong: not actually positive at all. There are several ways that an allegedly positive study can actually be negative …

• Flukes! Sometimes, by pure chance, things turn out well for the study subjects — just enough to make an ineffective treatment seem a little bit effective.
• Word trickery! Results are often described as “significant,” but this is statistical concept: trivial treatment effects can be statistically significant. And there are several other ways to summarize research to make it sound a little more promising than it really is.
• Lies! Many “positive” studies are simply worthless junk — true pseudoscience — published in ridiculous non-journals published by quacks with an self-promotional agenda. Such experiments have far too many fatal flaws to mean anything.
• More lies! It sounds incredible, but you’d be amazed how often people cite “positive” studies that just aren’t. If you read them, you end up scratching your head and thinking, “Um, but this is bad news … ”
• Honest mistakes! Even skilled, legitimate scientists have biases. There are many ways they can unconsciously bend experiments to produce the results they’d prefer to see, and/or they can spin the interpretation. Weakly positive results are often a symptom of this. Science is hard, and a lot of it is just wrong.

So you can see why I’m a little skeptical when someone enthusiastically shows me one paper from an obscure journal reporting a “significant” benefit to, say, acupuncture — which has probably been the subject of more of these “positive” studies than any other treatment.

This is an excerpt from a recent update to my article Most Pain Treatments Damned With Faint Praise: Most controversial and alternative therapies are fighting over scraps of “positive” scientific evidence that damn them with the faint praise of small effect sizes that cannot impress.