Overestimating “success rates” (for dry needling)
Today’s post is about how healthcare professionals are prone to overestimating their treatment success rates, using “dry needling” as a strong example. This began for me long ago, with a conversation I had circa 2005, in Vancouver, a hotbed of dry needling (there’s a doctor here who teaches a version of it, mostly to physiotherapists):
MASSAGE CLIENT: I had an awful time with dry needling. Total waste of cash, just caused a nasty flare-up of pain.
ME: Did you tell your physiotherapist?
CLIENT: Nope. I just didn’t go back. It was so awkward!
It’s astounding how often I have had variations of that talk over the years. But in that case — stuck in my head all these years — I also spoke to that therapist. He told me how his dry needling results were “always” great.
This is a thing therapists do: boast about their “success rate,” as if they can possibly know. They often include some faux humility — “of course it’s not perfect” — to avoid making it sound too good to be true. But it’s always still exaggerated.
Let’s back up a step…
Dry needling review
This post is not about acupuncture, but its weird “mainstream” cousin, “dry needling.” They look identical!
Dry needling is the insertion of acupuncture needles into sore spots associated with aching and stiffness (“trigger points”). It uses the needles of acupuncture, but not the principles. It is putatively pure biology, but rationalized with hey-you-never-know guesses about how those spots work, and why it might help to “stick ’em with the pointy end” (Arya Stark). Some of the guesses are fairly fancy, but they’re still guesses.
But who cares how it supposedly works, if it doesn’t actually work? The efficacy trials are disappointing reads (e.g. Stieven, Gattie).
Despite the studies, I suspect that some people truly are helped by dry needling — there do seem to be some “strong responders.” But the anecdotes also seem to skew to the extremes, love or hate. The clinical failures tend to be painful, even traumatic.
And mostly unreported.
People don’t like to give therapists bad news
Every therapist who has claimed “great results” for dry needling has probably greatly underestimated their failures. They mostly don’t even know about them!
Patient feedback is minimal at best, and distorted by several social forces and cognitive distortions, especially confirmation bias. Professionals, like all humans, are very strongly inclined to hear only what they want to hear. And clients are notoriously eager to please and often all too happy to tell them! They will bend the truth to make therapists happy, and to convince themselves that their time and money have not been wasted.
Even when outraged, people will still dodge and weave to avoid giving their therapist the bad news. They will even lie.
They tell me the bad news instead. They like telling me.
Every clinician has a nasty problem with “sampling bias” when estimating their own effectiveness. Clinicians simply don’t hear from the unhappiest patients, and from the rest they either hear too much of what they want and too little of what they don’t. Their “anecdata” is hopelessly skewed towards the flattering. (And this is why the whole idea of "intent to treat" exists in research.1)
And few understand how this works.
Buyer beware of “success rate” claims with dry needling — and anything else.
Dry needling is a topic I’ve written a lot about, but only behind a paywall. There’s a beefy chapter about it in my unique trigger points book — unique because it strikes a rare balance between credulity and skepticism on this tricky topic.
Confirmation bias is hard at work in every domain of life, but we are impressively oblivious to it most of the time. It has many effects on healthcare.
Notes
- “Intent to treat” is one of those bits of research jargon that makes something simple sound more exotic than it is. It’s really simple idea in principle: one of the main ways that we “control” research is by analyzing everyone that a treatment is intended to help, and not just the people that it actually did help. We want to know how everyone’s results from a well-controlled test — everyone that we try (intend) to treat, succeed or fail. By contrast, clinicians tend to only know about and consider the cases they *actually* (maybe) helped, the success stories — the hits rather than misses.