An anecdote about the trouble with anecdotes
Get posts in your inbox:
It’s hard to convince most people that anecdotes about medical treatments are unreliable. Maybe this embarrassing personal example will help: a story about how wrong these kinds of stories can be, an anecdote about the trouble with anecdotes.
I thought I had successsfully self-treated a problem. I won’t say what problem or treatment, because I don’t want to get caught up in the politics of the particulars. I had a symptom, and I did something for it; let’s call that attempt “X.”
It seemed like X made a difference, and — spoiler alert — I was probably wrong. How I got it wrong is what’s worth telling.
Before you can fall, you have to climb. I started to think optimistically about treatment X. I considered its implications for my website, for my readers, for my own life. I got my hopes up. I started to pet my theory. I knew it was speculative, and I didn’t get too carried away, but I was definitely starting to take it seriously. I did this for more than a day or two.
And then I realized my mistake.
Y and Z woke me up from my X dream
I was enjoying my idea so much that I had missed two other oddball variables — let’s call them Y and Z — that aren’t normally a part of my equation.
Y and Z were a bit far out in left field, and it’s just a bit of luck that I noticed them out there. I didn’t think my way to them. The only intelligence involved was in recognizing their significance. Something irrelevant made me notice Y, which made me wonder if I was missing anything else, which led me to Z, which led me to palm my face, because I realized that Y and Z could both independently explain my “success” with X. At the very least, Y and Z made it impossible to actually draw any conclusions. They broke the deal. Y and Z woke me up from my X dream.
In five minutes I went from thinking I was really onto something to thinking, “What was I thinking?”
It’s surprising that this would happen to me, of all people.
It’s like superman forgot about Kryptonite
I’ve been immersed in this kind of problem and subject matter for more than 15 years now, constantly trying to understand why people hurt, what helps and what doesn’t, and how we get it wrong. I’m a professional debunker, mythbuster, and truth seeker in the wilds of musculoskeletal medicine. And I came this close to missing two critical confounding variables, either one of which would have nuked an otherwise promising hypothesis.
And I almost blogged about it. 😱
But instead I’m writing this … because I realized just in time. By a fluke, basically.
If it can happen to me, it can happen to anyone
If I can make this mistake, how easy is it for someone unschooled in the ways of skepticism to fall into the same trap? Someone who has no idea what constitutes “critical thinking,” let alone in a medical context? Who doesn’t know the history of bizarre medical beliefs? Someone who can’t define a single logical fallacy and wouldn’t know a cognitive distortion if it bit them on the parietal lobe? Someone with a life?
Most people who tell treatment-success stories have only a single experience to draw on and know nothing else about that problem or the many other possible explanations for their relief (pain science is vast and strange).
This is why I can’t take it seriously when people tell me what they think works based on their personal experience. (Or even their professional experience.) I can’t take myself seriously, which is actually a central tenet of skepticism. Richard Feynman: “The first principle is that you must not fool yourself, and you are the easiest person to fool.”
The anecdote machine will keep churning out anecdotes
This story will not dent the overconfidence of people who believe they have experienced dramatic treatment successes.
When stubborn and serious problems appear to be cured within a few hour or days of receiving a new treatment, people will weld the two things together in their minds — and they are understandably irritated by any attempt to dismiss their story as an unreliable anecdote. Sadly, it is still an unreliable anecdote — not necessarily wrong, but still unreliable.
I tackle this sensitive topic in more detail in The Power of Barking: A silly metaphor for a serious point about correlation, causation, and how we decide what treatments work.