🇨🇦 🇨🇦
February 15 is National Flag of Canada Day. PainScience is a proudly Canadian small business, selling e-books and educational content about pain and injury since the mid-2000s. Read more about the project.
 
Detailed guides to painful problems, treatments & more • sitemap

Studying the Pain Studies

Tips and musings about how to understand (and write about) the extremely flawed science of pain and musculoskeletal medicine

Paul Ingraham • • 25m read

I am occasionally asked for advice on “how to do research” or “how to interpret studies” (this is usually how it’s framed) because PainScience.com has a reputation for diligence in this department — it’s one of the major distinguishing features of the site, along with an odd sense of humour and a peculiar obsession with salamanders. I read scientific studies and “translate” them for readers. I footnote my articles quite richly, where it counts, and I have a simply ginormous annotated bibliography.

I don’t do real “research,” of course. That’s for scientists. I hang out with scientists. I have beers with scientists. What I do is secondary research — not primary research in a lab, clinic, or the field. Secondary research is research about primary research.

Despite its second-class-citizen status, secondary research is an art and science in its own right, the core competency of the science journalist. There is not remotely a “right” way to do it — it’s so multi-faceted that I quite literally don’t know where to begin. So this page is a modest and random collection of thoughts and suggestions I’ve collected so far. I’ll add to it over time.

Those who don’t stay current with the literature think my perspective is contrarian.

Christopher Johnson, PT

DIY research analysis?

I never want to discourage anyone from trying, but I fear that it’s a mostly futile ambition — it’s just too hard. I have started backing away from encouraging people to try to interpret the science themselves, especially health science. It’s like encouraging people to do their own electrical repairs or colonoscopies. Some things you just really need help with. (Will you get good help? 🤷🏻‍♂️ Who knows! But that’s also a danger with electricians and gastroenterologists.)

it’s not impossible, of course. But most people will underestimate the challenge and never really get nerdy or serious enough about it to succeed. The learning curve is steep and endless, and almost has to be a full-time job. Even staying in my own lane, there are papers that I still can’t handle. If I leave my lane, I always want a guide: either a scientist who’s good at communication, or a journalist who’s good with the science.

The defining feature of most reporting on science is that it’s kind of terrible and half-arsed. So, if you do tackle the climb … aim higher!

Top journals for pain and injury science

Which scientific journals publish the most and best randomized controlled trials of physical therapy treatments? I wish I’d had this information a decade ago.

From the Department of Now You Tell Me, the journal Physical Therapy published an extremely useful set of reading recommendations for geeky, science-loving manual therapists.1 Costa et al. ranked journals by a variety of criteria, such as the sheer volume of relevant content and the prestige of the journal (“impact factor”). Using their lists of the top five for each category, it was easy for me to compile my own customized top ten list: I mashed up their various rankings into a single score, and then tweaked it to give greater weight to experimental quality and the subject matter which is of the greatest interest to me, my readers, and most manual therapists (musculoskeletal pain, rehab, etc).

The winners are …

  1. Archives of Physical Medicine and Rehabilitation
  2. Pain
  3. Physical Therapy
  4. Stroke
  5. Clinical Rehabilitation
  6. Spine
  7. Lancet
  8. British Medical Journal
  9. Journal of the American Medical Association

And the number one journal to read …

  1. Journal of Physiotherapy


Words of wisdom from the Physical Therapy article:

Physical therapists who are trying to keep up-to-date by reading the best available evidence on the effects of physical therapy interventions have to read more broadly than just physical therapy-specific journals. Readers of articles on physical therapy trials should be aware that high-quality trials are not necessarily published in journals with high impact factors.

Real professionals read journals

One of the places I dreaded most in graduate school was the “new journal desk” in the library, where all the science journals received the previous week were displayed, thousands of pages of them. Everyone would circle around it, teetering on the edge of panic attacks. All that available information seemed to taunt us with how out of control we felt—stupid, left behind, out of touch, and overwhelmed.

Robert M Sapolsky, Why Zebras Don’t Get Ulcers, 2004 p. 403

I’d like to briefly make the case for reading journals (or at least the best blogs about journals).

Millions of professionals around the world are regularly trying to help people with serious and chronic musculoskeletal pain, and yet most have never cracked open a medical journal — not even the ones most specialized and appropriate to their jobs. (Massage therapists and chiropractors in particular are guilty of this; personal trainers and naturopaths, too.) Most of those that do dabble in a little research reading tend to be skimmers and cherry pickers, looking through abstracts for quick confirmation of their own views. The result is usually bogus citations for their clinic blog.

The world is full of anti-intellectual and anti-scientific flakes, and they currently have little difficulty getting trained and licensed. My prescription for these professions is harsh: keep out the flakes. Gatekeeping with higher academic standards. If applicants can’t hack the science, they don’t get in the door, or they wash out early. Health care work should not be an option for amateurs with cute opinions like “I know what works” and “science doesn’t know everything.”2

Any serious healthcare profession knows this to be true. They have serious and credible entrance and competency standards for academic and clinical training. Professions that are don’t follow that model have always struggled for legitimacy and access. Having said that I think we should do more to help talented, smart and qualified LMTs, trainers and PTAs enter school and earn their DPT. I don’t think we do enough to facilitate that.

Jason Silvernail, DPT, DSc (online discussion)

Be one of those talented, smart professionals: read journals. But do it carefully. Because research shows that research is misleading!

Research shows research is misleading (and yet it’s surprising that the situation isn’t even worse)

Some of my readers often beg me to set my snarky, cranky, critical sights on Team Science for once. Okay. Here goes. (And it’s no big deal, of course, because science is quite neurotic and self-critical by nature. That’s why I love the big lug.)

Some French researchers went looking for misleading conclusions in scientific papers, and I’m sure you can guess where this is headed: nowhere good.3 As reported by Neil O’Connell for BodyInMind.org:

They looked for a selection of naughties: not reporting the results of the primary outcome, basing conclusions on secondary outcomes or the results of a sub-group analysis, presenting conclusions that are at odds with the data, claiming equivalence of efficacy in a trial not designed to test for it, and finally not considering the risk-benefit trade-off. Like a game of clinical trial Bullshit Bingo. They found evidence of misleading conclusions in 23% of reports. The only predictor of misleading conclusions was genuinely negative results. In trials with negative results the rate of misleading conclusions was, brace yourself, 45%.

Oh, is that all? Only 45%? I would have guessed about 70%. It sounds bad, but I really could not be any less surprised. Perhaps the conclusion is misleading?

This is hardly the first clue about this. Ioannidis famously taught us “Why Most Published Research Findings Are False” in 2005, even though the problem has been generally predictable since about 1795. And it’s obvious to nearly anyone who actually reads scientific papers, and not just their abstracts. That’s how I figured it out …

How I learned to distrust “conclusions” — a cautionary tale

Many moons ago, my journey towards the EBM light — that’s evidence-based medicine, if you haven’t been here before — began with a serious pain problem (iliotibial band syndrome) while I was training to be a massage therapist (an alarmingly evidence-free curriculum, alas). I had the important inspiration that, when the going gets serious, the serious go to the literature.

THE GREATEST THING EVER WRITTEN? In my books, yeah.

I really knew nothing about “the literature,” except that it was probably important. Indeed, all I really knew about science I’d learned from one book by Carl Sagan. I’d just finished reading Demon-Haunted World, which I was still fuming over, because I still wanted to believe in crop circles back then (no joke). It wasn’t until my second read a year later that I realized DHW was possibly THE GREATEST THING EVER WRITTEN.

So that was where I was as a “researcher” at the time that I actually started checking The Literature and reading scientific paper abstracts.

Abstracts were all I read, of course, and I took them all at face value. It was science! I was so impressed with myself just for using PubMed that it didn’t occur to me that abstracts are written by fallible people with agendas and funding worries and bosses and so forth. It was The Literature, dammit, and that was enough back then, and it was enough for me for at least a year. It was so easy! All I had to do was look shit up, and if an abstract had anything in it that sounded like it supported a point I was making, yahtzee! Make a footnote, instant credibility: citing science was like magic, ironically.

And then at some point I got particularly curious about some science and I actually read a whole scientific paper. I know, crazy, right? And guess what? The contents of that paper did not really square with the abstract. In fact, the paper’s “discussion” section and conclusion seemed distinctly at odds with the abstract.

Uh oh.4

Poop addendum: Neil wrapped up his description of this research by invoking a crap metaphor, as one does, to colourfully explain how the results of crap research often end up getting summarized in a way that makes them sound quite a lot better than they are: “You can’t polish a turd … but you can roll it in glitter.”5

Further down the rabbit hole

More highly relevant research also shows that study authors spin their own results — not just the media. Dr. Steve Novella wrote about it for Neurologica.

I have often had the impression that misleading spin in the discussion section of the paper is just as common as it is in the abstract — that is, the whole paper isn’t necessarily any better than the abstract. Recognizing that was just another step in the destruction of my innocence (1971–2003, RIP).

And it continued. Not only did I have to learn that the abstract doesn’t necessarily represent the paper, I also had to learn that the paper does not necessarily represent the data! And of course it goes deeper still, because the data may not represent reality. Down the rabbit hole!

Basically, there’s just an unbelievable number of ways trials can go wrong. Or be made to go wrong by “researchers” with virtually no real training in scientific methodology trying to prove their pet theories.6

That all might start to sound catastrophically embarrassing for science to some readers — is science hopeless? No: this is just why we never trust only one paper, or one source, and need lots (!) of replication and verification before getting cocky. “Proving” anything of any importance or complexity takes an enormous amount of checking, checking, checking from every angle for many years by many different researchers.

That’s just the job.

I used to rely on meta-analysis, but they are worse than laws & sausages, ceasing to inspire respect in proportion as we know how they are made.

Dr. Mark Crislip, "I Never Meta Analysis I Really Like"

PubMed Research Tip: use the clinical Queries feature

PubMed is the “Google” for medical scientific articles. It’s a service of the U.S. National Library of Medicine, that includes tens of millions of citations from MEDLINE and other life science journals for biomedical articles back to the 1950s. PubMed includes links to full text articles and other related resources.

I often use PubMed hundreds of times in a month. It’s a resource that all health care professionals should be at least a little bit familiar with. Patients can get some use out of it as well. However, it can be overwhelming!

Here’s a truly valuable PubMed tip: Use the Clinical Queries feature of PubMed. That produces search results that are much better for clinicians, especially non-medical professionals. The basic science stuff (all the microbiology, say) all gets filtered out … which is a lot more intelligible for those of us who don’t wear lab coats for a living.

Finding full text

Most scientific papers are paywalled — a major obstacle. This screenshot went wild on social media:

Screenshot of ironically expensive paywall for a scientific article titled “The growing ininaccessibility of science.”

Nature later took down the paywall on this (ancient) article — no doubt because of the flood of traffic! It’s easy to see why it went viral. A $200/year price tag on an article about the growing inaccessibility of science? That’s some chef’s kiss irony! ScienceAlert.com reports:

It’s not just the poor who don’t have access to scientific papers — journal subscriptions have become so expensive that leading universities such as Harvard and Cornell have admitted they can no longer afford them. Researchers have also taken a stand — with 15,000 scientists vowing to boycott publisher Elsevier in part for its excessive paywall fees.

However, as much as I’d like to see all science be free, I don’t actually think $200/year is an outrageous price for what Nature offers. And if any individual would pay that, wouldn’t it be someone exactly like me, a science journalist? But I don’t subscribe to most journals, because the articles I care about are scattered across far too many of them, despite my strong focus on pain.

The fees charged by most journals for temporary access to one article are usually well over USD $30; here’s one I wanted today that is, ouch, $68! Or $457 for the entire issue! Now that kind of pricing is absurd, and always has been.7 I do pay those fees occasionally, but less than I used to, because I have gotten better at getting free full text versions. For instance, just a few minutes after hitting that $68 paywall, I had the PDF directly from the lead author — because they can do that, and they are generally happy to.

So what are those options?

If you can’t get access to the full text, how much can you get from the abstract?

Almost every working day, I have to decide whether or not to pay for papers, or pirate them, or wait for them … and I routinely choose to settle for whatever I think I can safely wring out of the abstract. Scandalous? Is that enough? Can you really do justice to analysis of a scientific paper without reading the whole thing? Isn’t it a little like reviewing a movie by watching the trailer?

It is a little like that, yes. 😏 But you also didn't even need to see the trailer for Jurassic World: Dominion to know that it was going to be a garbage film. Trailers tell us some things quite clearly, and so do abstracts.

Not only is it possible to review papers based on the abstract, it’s just fine/necessary in many contexts. It’s just not possible to read all the relevant science (let alone pay for it). And the stakes are not equal for all citations. When citing to support particularly important and controversial points, don’t rely on just the abstract — but for many other citations, the abstract is certainly enough, if you do it cautiously.

Here are some mitigating considerations and tips for interpreting abstracts:

Comparing an abstract to a movie trailer is fun but poor. Abstracts have very different goals, and are not nearly as fragmentary and shallow as most trailers. In general, what actually matters in the paper is represented in the abstract.

Reader question: “Could you possible share a few ideas about how to stay current on relevant literature? What is your system?”

I use a lot of RSS feeds from journals, and scan those daily for relevant headlines, using good power tools, Reeder for Mac (an RSS reader) and Feedbin (a paid RSS subscription/syncing service and online reader). If you don’t know about RSS, see RSS in Plain English 5:00.

I also subscribe to the NEJM’s JournalWatch service which gives me good quality summaries.

But weirdly, social media is now probably the most important source of sources for me these days. I have a strong network of really smart colleagues who are also trying to stay current, and our collective efforts are extremely effective. As a group, we don’t miss much, and it’s often super clear what the most interesting recent papers are based on who shares what how often with how much enthusiasm. When a particularly intriguing and good quality paper is published, I’m going to find out!

So, cultivate virtual friendships with colleagues and mentors, and join Facebook discussion groups and fan pages where research gets discussed! They aren’t hard to find.

It’s a trap! Things to watch out for

Predatory journals. Since the 1990s, a shockingly high percentage of research is published without peer review, virtually guaranteeing poor quality. Even real peer review is imperfect, but when it’s missing entirely? •shudder• There is now an actual industry — a huge industry — of fraudulent journals that publish-for-pay.910 When evaluating a paper, always check for signs of a fake journal: no significant history, barely-there web presence, amateurish and sloppy, no impact factor, and so on.

“Active” controls. Many non-drug treatments are tricky to test scientifically, because it’s hard to compare them to a true fake treatment (like a comparing a drug to a sugar pill). Many scientists resort to comparing a treatment to some other treatment that we hope is “neutral” — maybe helpful, but not very helpful. Comparing a test treatment to an “active control” is likely to be misleading, because the active control pollutes the experiment with all kinds of unknown variables. These kinds of studies “need to be interpreted with caution.”11 It doesn’t mean that they are all wrong, but it’s a significant and basic limitation that has to be taken seriously. And “these kinds of studies” constitute a large percentage of studies in musculoskeletal medicine.

Statistical significance abuse is practically standard. There are two common forms. First, conflating "statistical significance" with "importance," often doing a victory dance based on only a nice p-value when the effect size is trivial. Second, boasting about the statistical significance of the wrong comparison, the amazingly basic but widespread error of only reporting within-group significance.

Bad science tends to make bad ideas look good. The reason pseudoscience exists is to seem like good science, and legitimize nonsense. There are many ways this happens, mostly bias and incompetence powering poor design, execution, analysis, and interpretation. But some of the problems are actually baked right into how science works: for instance, even if you were competent and objective, you'll tend to get false positives when you test scientifically implausible treatments, because of how statistics work (Pandolfi et al.). But even with these distortions, it’s difficult for low quality trials to make complete nonsense look effective — and so often what we see is results that are just barely positive. If a bunch of low quality trials of an implausible hypothesis cannot produce anything better than “promising” result, it’s almost certainly because there’s actually just nothing there. This applies to at least two thirds of all physical medicine research.

There’s a lot more, and I’ll add to this list slowly over time. Meanwhile, here are some other good sources:

Salamander logo, marking the end of the article.

About Paul Ingraham

Headshot of Paul Ingraham.

I am a science writer in Vancouver, Canada. I was a Registered Massage Therapist for a decade and the assistant editor of ScienceBasedMedicine.org for several years. I’ve had many injuries as a runner and ultimate player, and I’ve been a chronic pain patient myself since 2015. Full bio. See you on Facebook or Twitter., or subscribe:

Related Reading

I’ve written quite a lot over the years about science, research, and citing:

What’s new in this article?

Four updates have been logged for this article since publication (2013). All PainScience.com updates are logged to show a long term commitment to quality, accuracy, and currency. more Like good footnotes, update logging sets PainScience.com apart from most other health websites and blogs. It’s fine print, but important fine print, in the same spirit of transparency as the editing history available for Wikipedia pages.

I log any change to articles that might be of interest to a keen reader. Complete update logging started in 2016. Prior to that, I only logged major updates for the most popular and controversial articles.

See the What’s New? page for updates to all recent site updates.

Oct 3, 2025 — Beefed up the section on how to find full text scientific papers, and added more to the section about research analysis pitfalls.

2018 — Added tip about asking scientific paper authors for free full-text.

2018 — Started a new section: “It’s a trap! Things to watch out for.” Just a couple items to start.

2016 — Added a new section about how I find out about good new research.

2014, 2015 — A few unlogged updates.

2013 — Publication.

Notes

  1. Costa LOP, Moseley AM, Sherrington C, et al. Core Journals That Publish Clinical Trials of Physical Therapy Interventions. Phys Ther. 2010 Aug. PubMed 20724420 ❐

  2. “Science doesn’t know everything” is a classic, common non-sequitur from people defending quackery. It’s true but obvious, and irrelevant to their point … which is that their kooky treatment beliefs are so exotic that they are immune to investigation and criticism, beyond the reach of science. Nope! Not even close!

    It’s like declaring a leaky old canoe to be seaworthy because we don’t yet know everything about the depths of the ocean.

  3. Mathieu S, Giraudeau B, Soubrier M, Ravaud P. Misleading abstract conclusions in randomized controlled trials in rheumatology: comparison of the abstract conclusions and the results section. Joint Bone Spine. 2012 May;79(3):262–7. PubMed 21733728 ❐
  4. I remember that sinking feeling quite well. “Oh my,” I thought. “I’m going to have to have a closer look at an awful lot of papers. I’m going to be busy well into middle age!” Checking the literature got a great deal less sexy all at once. And now, fifteen years later — and just getting started on middle age — I am cynically surprised that the rate of misleading conclusions in abstracts is only a piddling 45%. Of course, it probably is much worse with the kind of research I mostly deal with: trials of complementary and alternative medicine, and a lot of obscure or dodgy or simplistic pet-theory cures for common pain problems.
  5. Fun imagery! But, oddly enough, you can polish a turd. It’s one of the more peculiar things the MythBusters have tested, demonstrating once again that you it’s important to check to see if reality actually works like you expect it to. Go Team Science!
  6. Cuijpers P, Cristea IA. How to prove that your therapy is effective, even when it is not: a guideline. Epidemiol Psychiatr Sci. 2016 Oct;25(5):428–435. PubMed 26411384 ❐
  7. I’m really not sure who the market for the full-price papers is, but the journals been pricing and marketing content like this consistently for a good twenty years now, through a period of great change in every kind of publishing. Either they are hopeless dinosaurs clinging to a stupid old way of doing things, leaving money on the table (totally plausible) … or they know something about their market and business that we don’t (maybe)… or a bit of both.

  8. Special pleading is an informal fallacy: claiming an exception to a general trend or principle without actually establishing that it is, either using a thin rationalization or even just using the exception as evidence for itself (“the rules don’t apply to my claim because my claim is an exception to the rule”).

  9. Beall J. What I learned from predatory publishers. Biochem Med (Zagreb). 2017 Jun;27(2):273–278. PubMed 28694718 ❐ PainSci Bibliography 52870 ❐

    ABSTRACT


    This article is a first-hand account of the author's work identifying and listing predatory publishers from 2012 to 2017. Predatory publishers use the gold (author pays) open access model and aim to generate as much revenue as possible, often foregoing a proper peer review. The paper details how predatory publishers came to exist and shows how they were largely enabled and condoned by the open-access social movement, the scholarly publishing industry, and academic librarians. The author describes tactics predatory publishers used to attempt to be removed from his lists, details the damage predatory journals cause to science, and comments on the future of scholarly publishing.

  10. Gasparyan AY, Yessirkepov M, Diyanova SN, Kitas GD. Publishing Ethics and Predatory Practices: A Dilemma for All Stakeholders of Science Communication. J Korean Med Sci. 2015 Aug;30(8):1010–6. PubMed 26240476 ❐ PainSci Bibliography 52903 ❐ “Over the past few years, numerous illegitimate or predatory journals have emerged in most fields of science.”
  11. Travers MJ, Bagg MK, Gibson W, O’Sullivan K, Palsson TS. Better than what? Comparisons in low back pain clinical trials. Br J Sports Med. 2018 Feb. PubMed 29420237 ❐

PainSci Member Login » Submit your email to unlock member content. If you can’t remember/access your registration email, please contact me. ~ Paul Ingraham, PainSci Publisher