PainSci summary of Colquhoun 2017?This page is one of thousands in the PainScience.com bibliography. It is not a general article: it is focused on a single scientific paper, and it may provide only just enough context for the summary to make sense. Links to other papers and more general information are provided at the bottom of the page, as often as possible. ★★★★☆4-star ratings are for bigger/better studies and reviews published in more prestigious journals, with only quibbles. Ratings are a highly subjective opinion, and subject to revision at any time. If you think this paper has been incorrectly rated, please let me know.
Dr. David Colquhoun briefly but persuasively argues that clinical guidelines and scientific reviews routinely make recommendations based on inadequate evidence, substantially due to a common failure to appreciate a the risk of false positives in positive studies of treatments with low prior plausibility: “every false positive not only harms patients (and budgets) but also provides ammunition for the antiscience brigade, who are now so evident.“
original abstract†Abstracts here may not perfectly match originals, for a variety of technical and practical reasons. Some abstacts are truncated for my purposes here, if they are particularly long-winded and unhelpful. I occasionally add clarifying notes. And I make some minor corrections.
Heneghan and colleagues discuss the need for better clinical guidelines. One problem with guidelines is that in very many cases no good treatment exists, yet recommendations are still made.
A good example is non-specific low back pain—the 2009 guidelines from the National Institute for Health and Care Excellence recommended acupuncture despite little evidence that it works to any useful extent. The guidelines were revised in 2016 to say “do not offer acupuncture.” This left few other recommended treatments, but the new guidelines fail to say explicitly that it’s an unsolved problem.
Even Cochrane reviews don’t seem to appreciate the distinction between P values and the risk of false positives. If you observe a P value close to 0.05, then to achieve a false positive rate of 5% you must assume that you are 87% certain of a real effect before the trial is done. That’s clearly preposterous. Even if you observe P=0.001, you would still have a false positive rate of 8% if the hypothesis was implausible (prior probability 0.1). And these numbers apply to perfect unbiased experiments, before you add all the other problems of P hacking, multiple comparisons, and so on.
These misunderstandings must be responsible for many false positive results. And every false positive not only harms patients (and budgets) but also provides ammunition for the antiscience brigade, who are now so evident.
- “The faulty statistics of complementary alternative medicine (CAM),” Maurizio Pandolfi and Giulia Carreras, Eur J Intern Med, 2014.
- “Clinical trials of integrative medicine: testing whether magic works?,” David H. Gorski and Steven P. Novella, Trends in Molecular Medicine, 2014.
- “Recommendations are made in the absence of any good treatments,” David Colquhoun, British Medical Journal, 2017.
One article on PainScience.com cites Colquhoun 2017 as a source:
- PS The “Impress Me” Test — Most controversial therapies are fighting over scraps of “positive” evidence that damn them with faint praise
This page is part of the PainScience BIBLIOGRAPHY, which contains plain language summaries of thousands of scientific papers & others sources. It’s like a highly specialized blog. A few highlights:
- A Bayesian model-averaged meta-analysis of the power pose effect with informed and default priors: the case of felt power. Gronau 2017 Comprehensive Results in Social Psychology.
- The neck and headaches. Bogduk 2014 Neurol Clin.
- Agreement of self-reported items and clinically assessed nerve root involvement (or sciatica) in a primary care setting. Konstantinou 2012 Eur Spine J.
- Effect of NSAIDs on Recovery From Acute Skeletal Muscle Injury: A Systematic Review and Meta-analysis. Morelli 2017 Am J Sports Med.
- Association of Spinal Manipulative Therapy With Clinical Benefit and Harm for Acute Low Back Pain: Systematic Review and Meta-analysis. Paige 2017 JAMA.