“It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.”
Sherlock Holmes (in Doyle’s 1892 A Scandal in Bohemia)
Here’s a very common error that I’ve heard countless times in online arguments and in angry emails:
“We need some science to prove how treatment modality X works!”
No! That is bass-ackwards. It’s a terrible inspiration for doing science, guaranteed to dial confirmation bias up to 11. But this idea is heard all too often in so-called “alternative” medicine. In fact, it’s one of its defining features: enthusiastic but amateurish speculation about the biological mechanisms of unproven treatments.
The premise that treatment X “works” is invariably based only on clinical experience and “educated” guessing, which proves nothing and cannot be trusted.
The fallibility of clinical experience: The pudding does not contain proof
“The proof is in the pudding.” I hear this almost daily from therapists who are convinced by their clinical experience that this or that treatment works. But the only thing in “pudding” is highly fallible clinical impressions, no better than any other form of anecdotal evidence, and possibly even worse. Working therapists in many ways are far more prone to biased misinterpretation and faulty perception than patients — they have ego, reputation, pride, and financial vested interests in being right.
But no: “proof” is only to be had from good clinical trials (and quite a few of them).
Efficacy trials first!
Treatment efficacy must be actually tested and demonstrated rigorously before trying to explain it. We need to find out if an idea actually produces measurable medical benefits and then — if it works, if it actually helps people — only then can we try to illuminate the mechanism.
We cannot study how a treatment works if we haven’t figured out if it works.
“Before we try to explain something, we should be sure it actually happened.”