“Promising” research never is

The word “promising” started backfiring for me about a decade ago, and I am now officially extremely fed up with “promising” research, because it never actually is. Or so close to never that there’s no practical difference.

I ❤️ this simple point from Hilda Bastian:

“A promising treatment is often in fact merely the larval stage of a disappointing one. At least a third of influential trials suggesting benefit may either ultimately be contradicted or turn out to have exaggerated effectiveness.”

Larval stage, snort. Very nicely put! That’s from the academic paper; see also Hilda’s more accessible post on the topic

The world has absurdly low standards for what constitutes “promising research” or a “growing body evidence” or what “research shows.” The only real requirement for optimism? Wanting something to be true.

It has gotten to the point where I just assume that any pain or physical medicine research touted as “promising” is almost certainly just a pathetically weak signal p-hacked out of a bunch of noise and spun into something worthy of a press release.

P.S. And no, even if a treatment really is promising — based on a genuine small benefit, rather than illusion of one — every little bit does not necessarily help.