Ironically, bad information is the chief hazard of the Information Age. The glut of unregulated information about your health available on the Internet1 constitutes both an incredible opportunity and a real threat to your safety and your pocketbook. (Even good information is hazardous!2) In the absence of any publication standards, how can you tell the difference between the good and the bad? There are three major signs of poor quality in health information:
Let’s look at those in some more detail …
A “claim” is any unverified assertion. But not all claims are created equal. Some claims matter more than others, and some are more dubious. In health care and health science, the stakes are high and many claims involve a lot of strikingly self-serving assertions (like “my Awesome Treatment Method™ will cure you”). If a claim could be used as a bullet-point in a sales pitch, it’s more claim-y. If it makes you (or your profession) look better, it’s more claim-y. And the claim-ier the claim, the more it needs to be backed up.
No one ever uses the word “claim” to describe their own beliefs and sales pitches, of course. Talking about claims is vocabulary for critical thinking and skepticism specifically, where it refers to a particular kind of claim — a special sense of the word — spawned by the thorny ethical challenges of selling care to sick, hurt people. Vulnerable people. (For instance, people suffering from aches and pains that are amazingly poorly understood for this late date in history.)
All claims need critical appraisal and independent verification, of course, but it’s much more important when it has more claim-stink and when people’s health depends on it. Sagan’s classic idea that “extraordinary claims require extraordinary evidence”3 is not just about exotic and/or extremely implausible claims like “aliens abducted me!” or “lake monsters are real” or “I have a machine that produces more energy than it requires, defying the laws of thermodynamics!” Really? Prove it! No one has trouble seeing those kinds of claims for what they are.
But the extraordinariness of claims is, in spirit, also about more mundane but profitable claims, such as “this nutritional supplement will reduce your knee pain” or “vibrating your tissues with sound waves will accelerate your tendon healing.” These are much more common hazards than truly exotic and extreme stuff, especially in health care. They are less obvious and yet more ethically problematic than many other common types of claims.
What can be asserted without proof can be dismissed without proof.
The history of medical science is full of examples of extraordinary claims that turned out to be correct.
History is much more full of extraordinary claims that went absolutely nowhere. To this day, several popular pseudosciences and superstitions persist on the fringes of society, and their advocates complain that they are not taken seriously — but they remain marginalized because they do not present extraordinary evidence, or any at all, to support their extraordinary claims.5History is full of extraordinary claims that went nowhere.
The most obvious kind of extraordinary claim in health care is the panacea, a treatment or medicine that is supposed to be good for anything that ails you. Here are some examples:
Panaceas are usually glaringly obvious. Unfortunately, health care is also full of extraordinary claims that are often mistaken for ordinary claims. A combination of widespread public biological illiteracy makes it easy for ignorant and/or deceitful people to make bad promises that seem perfectly reasonable to the average person. There is no easy defense against this except a high degree of caution.
Health care information and the profit (or prestige) motive are not usually good bedfellows.
However, it is not wrong in principle to profit from genuinely valuable health care services, products, or information. In fact, it’s necessary for those who have invested in training, product development or publication to promote themselves and earn a financial return — if what they sell is actually beneficial.
Unfortunately, it’s always been easy to make a profit from selling health care that is not beneficial, or only partially so. Public ignorance about science in general and biology in particular is epidemic. Worse still, people who are sick or hurt are eager to believe in anything that might help ease their pain, no matter how ridiculous or even dangerous. Since the only possible way to sell snake oil is to present deceitful and/or irrelevant information, there is an awful lot of that available.
The only solution for this is education — the buyer must be aware and critical. At the least, bogus health care products, services, and advice can waste your time and money. At worst, they may harm you. Never purchase unless you have been convinced by good quality reasoning and/or evidence. Don’t believe anything you are told by someone with a profit (or prestige) motive unless they support their claims responsibly.
Beware in particular of the most insidious threat of all: the salesman who persuades you to buy an ineffective product or service by giving you good information … about something else! Sneaky, eh? It happens all the time. Indeed, this is a favourite tactic of pharmaceutical manufacturers, who promote their reputations and increase sales of their products by providing perfectly good information to consumers and doctors that has nothing to do with the effectiveness or safety of their products. For example, they provide doctors’ with high quality patient education tools, such as charts — always with a logo on it.
The point is that you must never judge any health care product, service or advice by anything except the evidence for that product, service or advice — and the more someone stands to gain from your belief, the more careful you must be.
It is not just extraordinary claims that require extraordinary evidence — profitable claims require extraordinary evidence!
Unreferenced medical information is almost always useless. A health care publication without footnotes is like a doctor without a license to practice. When it comes to information about health, the stakes are simply too high for publishers to ask readers to take their word for anything.
Like references on a resumé it often doesn’t matter if you actually check citations — what matters is that it can be checked, and the more easily the better. Ideally, citations on the web should contains links to abstracts (summaries), or even the full text of the article. Recent publications should generally be preferred over older ones.A health care publication without footnotes is like a doctor without a license.
A nation’s currency rests on metal8 — there has to be real gold somewhere, or the value of our money is a mass delusion. In a similar sense, citations to scientific literature is the bedrock of medical information. Every medical idea, every recommended treatment, every new idea has to be — sooner or later — based on evidence published in a decent peer-reviewed scientific journal, and, if it can’t be, it had better have a good excuse. The greater the risks, the more extraordinary the claim, the more expensive the service … the greater the need for this trail of bread crumbs to the original research.
Not all health care information has to rise to this standard. Some of the articles I publish don’t require a lot of scholarship, for instance, because they ask nothing of the reader: they don’t have to buy anything, pick sides in a controversy, or make a medical decision based on the information. The need for evidence goes up with the stakes.
Unsupported medical opinions are sometimes accepable — the practice of health care would grind to a halt without them. But opinions are not evidence, must be clearly identified, and opinions that matter more must be more clearly identified. It’s irresponsible for an expert to disguise an opinion as fact, or to carelessly allow opinion to stand as fact in the minds of readers who may take expert opinions as seriously as good evidence. Experts must use cautious language when expressing opinions — such as “in my opinion” or “I believe.”
A document without references or acknowledgements of uncertainty is “information malpractice.”
You will be reasonably safe from bad information if you expect any significant claim to be supported by evidence, and if you raise the bar for extraordinary claims. If you are more cautious still when someone is trying to sell you something. And if you expect to see references at the end of any serious medical document. Do this, and you are officially a “critical reader.”
But there is one more important point, and it’s a good one to finish with.
There is no such thing as an unbiased or objective source of information,9 although some sources are more so than others. I believe that this is a myth perpetuated by journalists and profitable news media who want you to believe that they are a superior source of information. The truth is, everyone has a bias, and media consumers simply need to know what the bias is.
For instance, no one accuses Michael Moore or Rush Limbaugh of being unbiased — but they are good sources of information because we know what their biases are. The worst source of information is an unidentified source, where you do not know and cannot even guess at the biases.
Oddly enough, this is often an advantage of searching for information on the web: the biases of authors are often clear, and it’s often easy to find a wide variety of viewpoints quickly.The worst source of information is an unidentified source.
Please never be impressed by credentials alone. Credentials can be faked, and they often are. Although credentials may indicate expertise, and give you some idea of what the biases of the author may be, they do not guarantee good information. To be blunt, I believe that credentialled experts are often the worst of the “information offenders.” Unfortunately, it is a strong feature of human nature to rest on one’s laurels, and many professionals don’t seem to think that they need to support their claims.
Happy critical reading to you!
I am a science writer, former massage therapist, and assistant editor of ScienceBasedMedicine.org. I have had my share of injuries and pain challenges as a runner and ultimate player. My wife and I live in downtown Vancouver, Canada. See my full bio and qualifications, or my blog, Writerly. You might run into me on Facebook and Google, but mostly Twitter.
A scientific paper really is a different animal than a normal “article.” Most people have never read a scientific paper. Even though I have done a lot of homework in my field, I have probably only read a few dozen scientific papers at most.
What makes a scientific paper special is that there are strict standards for their publication, and clear general guidelines for all scientists about what constitutes good evidence. Publication in a scientific journal is not a guarantee that information is good — like democracy, it’s an imprefect system, but it’s the best we’ve got.
A good scientific paper has the following qualities:
So, now you are a seriously informed reader: the next time you see a dubious claim, look for supporting evidence published in a “peer-reviewed scientific journal.” Look it up. Is the paper “primary research”? Is so, was there a decent “sample”, and was it “randomized” and “controlled”?
Other interesting reading:
The following websites have little to do with health care specifically, but are interesting and essential resources for anyone generally interested in skepticism and critical thought:
An interesting debate has gone on … between those who think that all doctrines that smell of pseudoscience should be combated and those who believe that each issue should be judged on its own merits, but that the burden of proof should fall squarely on those who make the proposals. I find myself very much in the latter camp. I believe that the extraordinary should be pursued. But extraordinary claims require extraordinary evidence.
Broca's Brain, by Carl Sagan (p62, 1972 hardcover edition)
Carl Sagan was the Twentieth Century’s champion of public education about science, the scientific method, and critical thinking. His statement about extraordinary claims is perhaps the most famous of all comments ever made about skepticism, and it is the most important piece of wisdom an information warrior needs in the Twenty-First Century.BACK TO TEXT