Erroneous analyses of interactions in neuroscience: a problem of significance
One article on PainSci cites Nieuwenhuis 2011: Statistical Significance Abuse
PainSci commentary on Nieuwenhuis 2011: ?This page is one of thousands in the PainScience.com bibliography. It is not a general article: it is focused on a single scientific paper, and it may provide only just enough context for the summary to make sense. Links to other papers and more general information are provided wherever possible.
This research identified a major common problem in scientific papers. It was described by Ben Goldacre for The Guardian as “a stark statistical error so widespread it appears in about half of all the published papers surveyed from the academic neuroscience research literature.” Dr. Steven Novella also wrote about it for ScienceBasedMedicine.org recently, adding that “there is no reason to believe that it is unique to neuroscience research or more common in neuroscience than in other areas of research.”
original abstract †Abstracts here may not perfectly match originals, for a variety of technical and practical reasons. Some abstacts are truncated for my purposes here, if they are particularly long-winded and unhelpful. I occasionally add clarifying notes. And I make some minor corrections.
In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P> 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling.
related content
- “Why Most Published Research Findings Are False,” Ioannidis, PLoS Medicine, 2005.
This page is part of the PainScience BIBLIOGRAPHY, which contains plain language summaries of thousands of scientific papers & others sources. It’s like a highly specialized blog. A few highlights:
- Cannabidiol (CBD) products for pain: ineffective, expensive, and with potential harms. Moore 2023 J Pain.
- Inciting events associated with lumbar disc herniation. Suri 2010 Spine J.
- Prediction of an extruded fragment in lumbar disc patients from clinical presentations. Pople 1994 Spine (Phila Pa 1976).
- Characteristics of patients with low back and leg pain seeking treatment in primary care: baseline results from the ATLAS cohort study. Konstantinou 2015 BMC Musculoskelet Disord.
- Effectiveness and cost-effectiveness of universal school-based mindfulness training compared with normal school provision in reducing risk of mental health problems and promoting well-being in adolescence: the MYRIAD cluster randomised controlled trial. Kuyken 2022 Evid Based Ment Health.