PainScience.com Sensible advice for aches, pains & injuries
 
 

Extraordinary Health Claims

A guide to critical thinking, skepticism, and smart Internet reading about health care

updated (first published 2004)
by Paul Ingraham, Vancouver, Canadabio
I am a science writer and a former Registered Massage Therapist with a decade of experience treating tough pain cases. I was the Assistant Editor of ScienceBasedMedicine.org for several years. I’ve written hundreds of articles and several books, and I’m known for readable but heavily referenced analysis, with a touch of sass. I am a runner and ultimate player. • more about memore about PainScience.com

Ironically, bad information is the chief hazard of the Information Age. The glut of unregulated information about your health available on the Internet1 constitutes both an incredible opportunity and a real threat to your safety and your pocketbook. (Even good information is hazardous!2) In the absence of any publication standards, how can you tell the difference between the good and the bad? There are three major signs of poor quality in health information:

  1. extraordinary claims
  2. author hopes to sell you something
  3. no evidence or references offered (or bogus references)

Let’s look at those in some more detail …

Quackery Sign No. 1: Extraordinary claims (which are special in health care)

A “claim” is any unverified assertion. But not all claims are created equal. Some claims matter more than others, and some are more dubious. In health care and health science, the stakes are high and many claims involve a lot of strikingly self-serving assertions (like “my Awesome Treatment Method™ will cure you”). If a claim could be used as a bullet-point in a sales pitch, it’s more claim-y. If it makes you (or your profession) look better, it’s more claim-y. And the claim-ier the claim, the more it needs to be backed up.

No one ever uses the word “claim” to describe their own beliefs and sales pitches, of course. Talking about claims is vocabulary for critical thinking and skepticism specifically, where it refers to a particular kind of claim — a special sense of the word — spawned by the thorny ethical challenges of selling care to sick, hurt people. Vulnerable people. (For instance, people suffering from aches and pains that are amazingly poorly understood for this late date in history.)

All claims need critical appraisal and independent verification, of course, but it’s much more important when it has more claim-stink and when people’s health depends on it. Sagan’s classic idea that “extraordinary claims require extraordinary evidence3 is not just about exotic and/or extremely implausible claims like “aliens abducted me!” or “lake monsters are real” or “I have a machine that produces more energy than it requires, defying the laws of thermodynamics!” Really? Prove it! No one has trouble seeing those kinds of claims for what they are.

But the extraordinariness of claims is, in spirit, also about more mundane but profitable claims, such as “this nutritional supplement will reduce your knee pain” or “vibrating your tissues with sound waves will accelerate your tendon healing.” These are much more common hazards than truly exotic and extreme stuff, especially in health care. They are less obvious and yet more ethically problematic than many other common types of claims.

What can be asserted without proof can be dismissed without proof.

Christopher Hitchens

The history of medical science is full of examples of extraordinary claims that turned out to be correct.

History is much more full of extraordinary claims that went absolutely nowhere. To this day, several popular pseudosciences and superstitions persist on the fringes of society, and their advocates complain that they are not taken seriously — but they remain marginalized because they do not present extraordinary evidence, or any at all, to support their extraordinary claims.5

History is full of extraordinary claims that went nowhere.

The most obvious kind of extraordinary claim in health care is the panacea, a treatment or medicine that is supposed to be good for anything that ails you. Here are some examples:

Panaceas are usually glaringly obvious. Unfortunately, health care is also full of extraordinary claims that are often mistaken for ordinary claims. A combination of widespread public biological illiteracy makes it easy for ignorant and/or deceitful people to make bad promises that seem perfectly reasonable to the average person. There is no easy defense against this except a high degree of caution.

Quackery Sign No. 2: Selling (ka-ching, ka-ching!)

Health care information and the profit (or prestige) motive are not usually good bedfellows.

However, it is not wrong in principle to profit from genuinely valuable health care services, products, or information. In fact, it’s necessary for those who have invested in training, product development or publication to promote themselves and earn a financial return — if what they sell is actually beneficial.

Unfortunately, it’s always been easy to make a profit from selling health care that is not beneficial, or only partially so. Public ignorance about science in general and biology in particular is epidemic. Worse still, people who are sick or hurt are eager to believe in anything that might help ease their pain, no matter how ridiculous or even dangerous. Since the only possible way to sell snake oil is to present deceitful and/or irrelevant information, there is an awful lot of that available.

The only solution for this is education — the buyer must be aware and critical. At the least, bogus health care products, services, and advice can waste your time and money. At worst, they may harm you. Never purchase unless you have been convinced by good quality reasoning and/or evidence. Don’t believe anything you are told by someone with a profit (or prestige) motive unless they support their claims responsibly.

Beware in particular of the most insidious threat of all: the salesman who persuades you to buy an ineffective product or service by giving you good information … about something else! Sneaky, eh? It happens all the time. Indeed, this is a favourite tactic of pharmaceutical manufacturers, who promote their reputations and increase sales of their products by providing perfectly good information to consumers and doctors that has nothing to do with the effectiveness or safety of their products. For example, they provide doctors’ with high quality patient education tools, such as charts — always with a logo on it.

The point is that you must never judge any health care product, service or advice by anything except the evidence for that product, service or advice — and the more someone stands to gain from your belief, the more careful you must be.

It is not just extraordinary claims that require extraordinary evidence — profitable claims require extraordinary evidence!

Quackery Sign No. 3: No evidence, no references = information malpractice

Unreferenced medical information is almost always useless. A health care publication without footnotes is like a doctor without a license to practice. When it comes to information about health, the stakes are simply too high for publishers to ask readers to take their word for anything.

Like references on a resumé it often doesn’t matter if you actually check citations — what matters is that it can be checked, and the more easily the better. Ideally, citations on the web should contains links to abstracts (summaries), or even the full text of the article. Recent publications should generally be preferred over older ones.

A health care publication without footnotes is like a doctor without a license.

A nation’s currency rests on metal8 — there has to be real gold somewhere, or the value of our money is a mass delusion. In a similar sense, citations to scientific literature is the bedrock of medical information. Every medical idea, every recommended treatment, every new idea has to be — sooner or later — based on evidence published in a decent peer-reviewed scientific journal, and, if it can’t be, it had better have a good excuse. The greater the risks, the more extraordinary the claim, the more expensive the service … the greater the need for this trail of bread crumbs to the original research.

Not all health care information has to rise to this standard. Some of the articles I publish don’t require a lot of scholarship, for instance, because they ask nothing of the reader: they don’t have to buy anything, pick sides in a controversy, or make a medical decision based on the information. The need for evidence goes up with the stakes.

Unsupported medical opinions are sometimes accepable — the practice of health care would grind to a halt without them. But opinions are not evidence, must be clearly identified, and opinions that matter more must be more clearly identified. It’s irresponsible for an expert to disguise an opinion as fact, or to carelessly allow opinion to stand as fact in the minds of readers who may take expert opinions as seriously as good evidence. Experts must use cautious language when expressing opinions — such as “in my opinion” or “I believe.”

A document without references or acknowledgements of uncertainty is “information malpractice.”

A final quackery consideration: the source

You will be reasonably safe from bad information if you expect any significant claim to be supported by evidence, and if you raise the bar for extraordinary claims. If you are more cautious still when someone is trying to sell you something. And if you expect to see references at the end of any serious medical document. Do this, and you are officially a “critical reader.”

But there is one more important point, and it’s a good one to finish with.

There is no such thing as an unbiased or objective source of information,9 although some sources are more so than others. I believe that this is a myth perpetuated by journalists and profitable news media who want you to believe that they are a superior source of information. The truth is, everyone has a bias, and media consumers simply need to know what the bias is.

For instance, no one accuses Michael Moore or Rush Limbaugh of being unbiased — but they are good sources of information because we know what their biases are. The worst source of information is an unidentified source, where you do not know and cannot even guess at the biases.

Oddly enough, this is often an advantage of searching for information on the web: the biases of authors are often clear, and it’s often easy to find a wide variety of viewpoints quickly.

The worst source of information is an unidentified source.

Please never be impressed by credentials alone. Credentials can be faked, and they often are. Although credentials may indicate expertise, and give you some idea of what the biases of the author may be, they do not guarantee good information. To be blunt, I believe that credentialled experts are often the worst of the “information offenders.” Unfortunately, it is a strong feature of human nature to rest on one’s laurels, and many professionals don’t seem to think that they need to support their claims.

Happy critical reading to you!


About Paul Ingraham

Headshot of Paul Ingraham, short hair, neat beard, suit jacket.

I am a science writer, former massage therapist, and I was the assistant editor at ScienceBasedMedicine.org for several years. I have had my share of injuries and pain challenges as a runner and ultimate player. My wife and I live in downtown Vancouver, Canada. See my full bio and qualifications, or my blog, Writerly. You might run into me on Facebook or Twitter.

Appendix: What makes a scientific paper so scientific?

A scientific paper really is a different animal than a normal “article.” Most people have never read a scientific paper. Even though I have done a lot of homework in my field, I have probably only read a few dozen scientific papers at most.

What makes a scientific paper special is that there are strict standards for their publication, and clear general guidelines for all scientists about what constitutes good evidence. Publication in a scientific journal is not a guarantee that information is good — like democracy, it’s an imprefect system, but it’s the best we’ve got.

A good scientific paper has the following qualities:

So, now you are a seriously informed reader: the next time you see a dubious claim, look for supporting evidence published in a “peer-reviewed scientific journal.” Look it up. Is the paper “primary research”? Is so, was there a decent “sample”, and was it “randomized” and “controlled”?

Related Reading

Other interesting reading:

The following websites have little to do with health care specifically, but are interesting and essential resources for anyone generally interested in skepticism and critical thought:

Notes

  1. In 2010, the Journal of Bone & Joint Surgery reported that “the quality and content of health information on the internet is highly variable for common sports medicine topics,” such as knee pain and low back pain — a bit of an understatement, really. Expert reviewers examined about 75 top-ranked commercial websites and another 30 academic sites. They gave each a quality score on a scale of 100. The average score? Barely over 50! For more detail, see Starman et al. BACK TO TEXT
  2. Googling your symptoms is as unavoidable as it is unwise. We almost literally cannot help ourselves. I imagine there is a modern epidemic of reckless, panicky self-treatment based on really bad Internet-powered self-diagnosis — something I am acutely aware of every time I hit my magic “publish” button. (With great power…) The Flemish government is acutely aware of this problem too, which is why they commissioned a couple of (hilarious) public service announcements about it, one that’s pure sketch comedy, and one about their clever delivery method: the clever monkeys used Google’s own advertising program to target people searching for common symptoms! BACK TO TEXT
  3. An interesting debate has gone on … between those who think that all doctrines that smell of pseudoscience should be combated and those who believe that each issue should be judged on its own merits, but that the burden of proof should fall squarely on those who make the proposals. I find myself very much in the latter camp. I believe that the extraordinary should be pursued. But extraordinary claims require extraordinary evidence.

    Broca's Brain, by Carl Sagan (p62, 1972 hardcover edition)

    Carl Sagan was the Twentieth Century’s champion of public education about science, the scientific method, and critical thinking. His statement about extraordinary claims is perhaps the most famous of all comments ever made about skepticism, and it is the most important piece of wisdom an information warrior needs in the Twenty-First Century.

    BACK TO TEXT
  4. John Snow was the father of epidemiology, the science of diseases and how they spread. In 1854, Snow mapped cases of cholera and traced the disease to its origin, and proved with a simple test that it was coming from a polluted well: he removed the pump handle, and the cholera outbreak stopped. Naturally, this conclusion was scoffed at by those with competing theories (“bad air”), producing one of my favourite history of medicine stories: to prove that cholera was not in the water, some smartypants biologist drank a glass of the stuff … and got away with it, because he just happened to be immune. It’s a great example of why “experience” is not trustworthy, and why it takes so long to settle even relatively straightforward puzzles in medicine. BACK TO TEXT
  5. Consider the example of Harry Houdini, who sincerely wanted a medium to help him talk to his dead mother — but the master of illusion could not find a medium who was not an obvious fraud to him. Or, consider James Randi, a prominent skeptic and debunker, who has offered a million-dollar prize for the last several years to anyone who can convincingly demonstrate a paranormal ability such as ESP, telekinesis or clairvoyant interpretation of Tarot cards. No challenger has yet won the prize. BACK TO TEXT
  6. PS Ingraham. Does Chiropractic Work? An introduction to chiropractic controversies like aggressive billing, spinal adjustment as a panacea, treating kids, neck manipulation risks, and more. PainScience.com. 6925 words. BACK TO TEXT
  7. PS Ingraham. Water Fever and the Fear of Chronic Dehydration: Do we really need eight glasses of water per day? PainScience.com. 3802 words. BACK TO TEXT
  8. Well, that’s how it started anyway! The gold standard was abandoned by all nations. All currencies are now fiat — that is, their value is basically just declared by the government, so modern money is based on confidence in the government (and purchasing power) rather than gold. And yet, critically for my analogy, most countries do maintain substantial gold reserves! Probably not for nothing. BACK TO TEXT
  9. PS Ingraham. Objectivity is Overrated: A response to the common accusation of bias and the mythical virtue of objectivity. PainScience.com. 445 words. BACK TO TEXT