Researchers examined 196 trials of psychotherapy for depression. Only a third of the studies found that therapy was helpful. Of those finding therapy unhelpful, 94% reported their results in a misleading manner (known as “spin”) to make the treatment look more helpful than it was.
Additionally, 76% of the studies reported conclusions based on different measures than they had initially planned to use, an unethical practice known as “outcome reporting bias” or “outcome switching.”
“This study shows a high prevalence of protocol discrepancies and spin and low rates of registered trials in psychotherapy research,” the researchers write.
The research was led by Marlene Stoll at University Medical Center Mainz, Germany, and published in the Journal of Clinical Epidemiology. Stoll et al. focused on a single diagnosis, depression, and on trials published between 2015 and 2018, as a way of getting a snapshot of the current state of psychotherapy research for a single, prevalent, well-studied diagnosis.
Of the 196 trials of therapy for depression, a mere 34% found a positive effect of their therapy intervention. 39% found that therapy did not help, and 27% had unclear or mixed results. Worse, 94% of the studies that found therapy unhelpful spun their results to make therapy look better. Only four trials did not spin their results.
What did spin look like?
On average, each paper spun their results about five times (or in five different ways). The most common forms of spin were as follows:
- The researchers didn’t mention in the abstract that the primary outcome was non-significant (54%)
- The researchers claimed that the treatment was beneficial despite non-significant findings in the abstract (43%) and the conclusion section (46%)
Another major form of spin that pervaded the papers was distracting the reader by writing about other outcomes before eventually admitting that the primary outcome was non-significant.
Outcome Reporting Bias
Perhaps even more concerningly, 76% of the studies reported outcomes that were inconsistent with their registered trial protocols—which means even the trials that supposedly found a positive effect for therapy may be untrustworthy.
To prevent misleading reporting of results, the World Medical Association’s 2013 Declaration of Helsinki requires that researchers submit a trial protocol before beginning their study. Unfortunately, not every researcher adheres to the Declaration of Helsinki’s guidelines. In this case, 32% of the trials did not have a registered trial protocol.
Registering a trial protocol means that the researcher is telling the reader the specific outcome measures they’re going to use. This protocol remains online (at clinicaltrials.gov, for instance). That way, if researchers report a different set of outcome measures, it can be identified, and the reader knows that the research is untrustworthy.
Why would researchers change their outcome measures? Well, according to Stoll et al., researchers are encouraged to find a positive effect of their treatment. The study sponsor may want them to find a positive effect (this occurs most often in pharmaceutical trials), but researchers are also invested in finding a positive effect themselves. It makes publication a lot more likely, and that’s the biggest metric of a researcher’s career success (and can mean more grant money).
Moreover, researchers are not unbiased. Psychotherapy researchers may be desperate to prove that their mode of therapy is better than the alternative.
So, for instance, researchers who hope to prove that their treatment is effective for depression might use several different depression measures, such as the Hamilton Rating Scale for Depression (HAM-D) or the Beck Depression Inventory (BDI). The more testing you do, the more likely you are to find some difference between groups, just by chance. If the researchers don’t find an effect on the HAM-D, but they do find an effect on the BDI, they may surreptitiously choose only to report the positive finding.
Supposedly, registering trial protocols prevents this practice. But, as Stoll et al. report, 76% of the registered trials still engaged in outcome switching.
What’s the Solution?
Stoll et al. suggest that to prevent these unethical and misleading practices, several steps should be taken.
First, journals must demand that trial protocols be registered, and someone in the editorial process must check for outcome switching.
Second, researcher affiliation must be better understood. That is, we can expect that researchers who support a particular therapy are going to try to make their modality look better.
Third, “registered reports” should be the standard publication strategy moving forward. In this paradigm, researchers submit the methods for their study before conducting the study. The journal then accepts or rejects their paper based on their methods. That way, even if the researchers find a negative result, their study will still be published. This decreases publication bias; currently, only positive results are regularly published.
Stoll M, Mancini A, Hubenschmid L, Dreimüller N, König J, Cuijpers P, Barth J, & Lieb K. (2020). Discrepancies from registered protocols and spin occurred frequently in randomized psychotherapy trials – a meta-epidemiologic study. Journal of Clinical Epidemiology. DOI: https://doi.org/10.1016/j.jclinepi.2020.08.013. (Link)