Wim Hof Method vs. The Null Hypothesis

I took more cold showers this year than I did for my entire life before that — and probably the rest of it, too.

Why would I do such a thing? To reduce inflammation, naturally. Hopefully. I was mostly inspired by a study. The showers were actually a sideshow: I was really in it for the breathing, because the evidence showed some anti-inflammatory potential for that (hyperventilating specifically), while cold didn’t actually move the needle (and neither did expert coaching). But I wanted to do both, partly because I knew the combo is similar to the famous Wim Hof Method (WHM), the popular cocktail of breathin’, meditatin’, and chillin’.

And I just get a kick out of (safely) treating myself like a lab rat. I don’t mind a little sensory drama “for science.” It’s almost part of my job!

I thought the showers were a species of fun. That’s probably all they were: transient physiological invigoration, an amusing way to just feel quite different … but who couldn’t use a bit of that now and then? I felt alive after those showers! For a good twenty minutes! Maybe even thirty!

And then, every time, I slipped back into the usual, thinking about how easy it would be to confuse “invigoration” with “real health benefits.”

Fountain of health?

The first real test of Wim Hof Method

Today’s post is about the first proper, direct test of the Wim Hof Method, by Ketelhut et al. Several studies have been WHM adjacent, and its ingredients have been studied independently quite a bit (especially meditation), but no one has yet studied Hof’s recipe, the Iceman’s own protocol (the Ice-meister, the Ice-inator, makin’ copies…).

And many are saying that they still haven’t! But that was certainly the idea of the study.

The official WHM has many touted health benefits, and has catapulted to fame based on Hof’s genuinely remarkable cold tolerance (which, importantly, may well be genetic, not trained). If the WHM does what it says on the tin, then it shouldn’t be hard to prove it. And if it is hard to prove it, how good can it be? So just get some people to make a good faith effort for a couple of weeks, and they should show some signs of boosted health.

People aren’t going to turn into superhumans that fast, not even if WHM is genuinely effective. But they should show some signs.

They didn’t in this test. It wasn’t just a case of “technically positive but not impressive” (which I routinely round down to “negative”). No statistically significant benefits were detected at all.

Unsurprisingly, skeptics are smugly sharing the bad news, while the WHM faithful are screaming, “Objection!” The reactions are as perfect an example as we’ve ever seen of how enthusiastically people nitpick studies they don’t like, or avoiding nitpicking the ones they do. Here’s a reminder about how this works (I really get a lot of use out of this flowchart these days):

This study is not a huge win for Wim Hof skeptics

Perhaps a battle, but maybe not even that, and certainly not the war. As negative as this trial of the Wim Hof Method was, it’s just not quite the satisfying game-ender of a slam dunk that some skeptics seem to think it is. There are many good reasons to be skeptical of the Iceman’s claims… but not yet conclusive evidence of absence.

This trial is just not enough. For many reasons (cultural and scientific), you cannot put a topic like this to rest without extremely robust evidence. Which this is clearly not.

I’ve seen several commentators mistakenly identifying it as a study in the journal Nature, which would certainly boost its credibility. But nope! Although it’s on Nature.com, this paper is not published by “the” journal Nature — it’s their huge open-access mega-journal subsidiary, Scientific Reports, which has some quality control and credibility issues — although many papers are just fine. So no prestige boost.

The most legitimate gripe about the study is that we don’t know just how frigid those showers actually were… because none of this went down in a Level 3 Shower Lab. The authors acknowledge this problem, and there are some reasonable reasons why it’s only a “limitation” and not a deal-breaker… but clearly someone should do this again with more exactingly shivery showers.

So the study just can’t be nail-in-coffin material. Even if it was seemingly perfect, it would still only be one negative study: discouraging, but not definitive.

Meanwhile, the evidence is already a bit tricky, because the components of WHM have been tested in so many other ways, sometimes even in very Hof-ian combinations. Zwaag et al was really quite close, and genuinely encouraging … about breathing. But specifically not about cryotherapy — which is obviously the heart of the Iceman's method. (For my full report on that study, see “Anti-inflammatory hyperventilation: I’ll huff and I’ll puff and I’ll blow my pain away”.)

But the study is credibly discouraging (despite much scoffing from the WHM camp)

Ketelhut et al may not have “proved” that WHM is bogus, but it also was not the doomed-to-fail dumpster fire of a trial that every WHM apologist is hollering about.

There’s a strong theme in the objections: it wasn’t true WHM’ing. Not good enough meditating, not good enough breathing, and definitely not good enough cryotherapy (the showers of uncertain coldness).

But this study was never intended to be a study of perfect WHM’ing. It was clearly conceived as a more realistic study of the kind of WHM that the average person can realistically be expected to achieve short of showering with Wim Hof himself. Mr. Hof has been clear that ordinary people can get these benefits quickly and easily — that is part of the legend and the marketing. (Importantly, we also already know from Zwaag et al. that more advanced training with teachers was not superior.)

Despite the fact that it wasn’t a test of the best possible WHM, it was roughly as good as you could ever expect from the average person.

And if a modest, achievable dose of WHM gets you zero benefit, then it’s just too hard to be useful for most people, and it’s directly at odds with how it is being advertised.

In other words, if you have to be hardcore to get a benefit, most people cannot get it.

This is far from “proof” that WHM doesn’t work. Being hardcore might actually work, for instance! But it is a legitimately discouraging result.

Never bet against the null

Wim Hof does not have a monopoly on breathing exercises, meditation, and cryotherapy: they are each popular in their own right, and there are good reasons for skepticism about each of them.

Hof is promoting the idea of a just-right cocktail of these practices. It’s plausible that his blend is effective, but it’s not particularly likely given the generally poor scientific track record of the independently tested ingredients.

Never bet against the null hypothesis (“there is no effect”). In every trial, The Null is like a champion boxer in the other corner of the ring, menacing and extremely hard to take down, waiting to thump the hypothesis. Sure, sometimes the hypothesis wins… but rarely. And that’s the point.

I am not planning on taking many more cold showers in my life either. But I’ll probably do it more often than I bet against the null.