PATIENT’S FREEDOM OF INFORMATION REQUEST TORPEDOES RECOVERY CLAIMS IN £5 MILLION TRIAL OF EXERCISE THERAPY FOR CHRONIC FATIGUE SYNDROME
The researchers’ published results have been overturned by expert patients, backed by statisticians, applying the study’s originally planned analyses
Wednesday, 21st September, 2016, London, UK — Data obtained by a patient under the Freedom of Information Act has contradicted published claims that exercise and psychotherapy helped people with chronic fatigue syndrome to recover in the £5 million PACE trial.
The results follow years of criticism from patients who pointed out a range of problems in the government-funded study, including that key analyses had been altered and that patients could be classed as recovered even if they became more disabled during the trial.
Now, in an extraordinary example of patients intervening in matters of their own health, patients with mathematical expertise have used the newly released data to calculate recovery rates using the study’s originally specified methods. Statistics professors Bruce Levin of Columbia University, New York, and Philip B. Stark of the University of California, Berkeley, confirmed their results.
In contrast to the study authors’ claims of recovery rates of 22%, the new analyses found that only 4% of patients who received exercise therapy, and 7% who received cognitive behavioural therapy (CBT) recovered, while 3% of a no-therapy comparison group did so.
Alem Matthees, a patient from Perth, Australia, who obtained the data and led the analysis, said that the differences between groups were within the range of chance variation, and indicated no benefit of the therapies.
The journals that reported PACE’s findings have so far not acted on complaints about the published analyses, which include an open letter from over 40 scientists and doctors stating that the flaws in the trial “have no place in published research”.
Dr David Tuller, a journalist and public health expert at the University of California, Berkeley, who has written extensively on problems in the PACE trial, said, “Now that these claims have been exposed as nonsense by an analysis of the actual trial data, who is going to be held accountable for the £5 million wasted on this badly done and uninterpretable piece of research?”
Professor Levin said, “Respect for the journals who published PACE's findings — The Lancet and Psychological Medicine — has been diminished worldwide by their position defending this trial. Retraction of the flawed analyses would be appropriate”.
Professor Jonathan Edwards, of University College London, said that the PACE study appeared to provide substantial evidence for lack of usefulness of the therapies, and that this should now inform UK treatment guidelines. Currently, NICE recommends the PACE therapies as the main treatments for hundreds of thousands of chronic fatigue syndrome patients in the UK. The therapies are also widely recommended abroad.
Expert reaction to the new analyses
Patient and lead author on the new analysis, Perth, Australia
The problems with PACE were clear as soon as it was published in The Lancet and then later in Psychological Medicine, yet for years, patients’ criticisms about very serious flaws have been ignored. Something has gone wrong with the quality control and the peer-review process in relation to this trial, and now questions should be asked about how this happened.
(The following statement, made with co-authors, is from the analysis summary.)
The PACE trial provides a good example of the problems that can occur when investigators are allowed to substantially deviate from the trial protocol without adequate justification or scrutiny. We therefore propose that a thorough, transparent, and independent re-analysis be conducted to provide greater clarity about the PACE trial results.
Pending a comprehensive review or audit of trial data, it seems prudent that the published trial results should be treated as potentially unsound, as well as the medical texts, review articles, and public policies based on those results.
David Tuller, DrPH
Lecturer in public health and journalism, University of California, Berkeley
These findings confirm what was self-evident — the results they reported were much, much better than the results they actually got using the methods they had promised to use in their protocol. As was obvious from reading the papers, the changes made it significantly easier for them to declare success and prove "recovery." Now that these claims have been exposed as nonsense by an analysis of the actual trial data, who is going to be held accountable for the £5 million wasted on this badly done and uninterpretable piece of research?
Bruce Levin, PhD
Professor and Past Chair, Department of Biostatistics, Columbia University
The originally planned analyses that have now been conducted show little evidence that CBT and graded exercise therapy add anything to standard medical care in terms of patient recovery. The differences between the groups in the trial are well within chance variation.
Using criteria for recovery that would allow for no real improvement was amateurish. The investigators reported that the trial steering committee approved the changes to the analyses, but a steering committee is generally not the same as an independent data monitoring committee, charged with assuring patient safety and trial integrity. In particular, data monitoring committees, as opposed to steering committees, generally exclude the investigators.
Respect for the journals who published PACE's findings — The Lancet and Psychological Medicine — has been diminished worldwide by their position defending this trial. Retraction of the flawed analyses would be appropriate, but even more important would be for the journals to publish further analyses allowing for full and open discussion of what happened in PACE.
Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine, University College London
The reanalysis by Matthees and colleagues shows that the call for access to the PACE data for independent analysis was justified. Together with the reanalysis of outcome recently made by the PACE authors and the follow-up study published late last year, it confirms that the PACE study failed to provide reliable evidence for useful, sustained benefit from either CBT or graded exercise therapy.
Perhaps more importantly, the PACE study appears to provide substantial evidence for lack of usefulness of these therapies, at least in the forms used in the trial. This should now inform UK treatment guidelines (a reassessment has already been made in the USA). The original PACE publications should at least be modified to carry an addendum to indicate the subsequent reanalyses.
Ronald W. Davis
Professor of Biochemistry & Genetics, and Director of the Stanford Genome Technology Center at Stanford University
This new analysis of the original data from the PACE trial indicates that the tested treatments have no benefits. It is also widely known that exercise can do great harm to ME/CFS patients. Therefore, this paper should be retracted by the authors and/or the journal. Furthermore, no doctor should continue to practice the paper's teachings. It is essential that everyone follow the oath "DO NO HARM!"
Rebecca Goldin, PhD
Professor of Mathematical Sciences, George Mason University, USA and Director, STATS.org
Small differences that are not "statistically significant" suddenly reach statistical significance when differences are scaled up, even if all numbers are increased proportionally. The changing recovery criteria had exactly this effect on the PACE data analysis. Weakening the recovery criteria boosts the recovery rates for all treatments, magnifying small differences in patient outcome. Clinically inconsequential differences, consistent with the view that all treatments were equally beneficial, suddenly balloon into statistically significant differences among trial arms. Statistically significant differences may have become clinically significant in some people's minds, since the purported benefits pertained to "recovery" rates. The fact that the very definition of "recovery" may have determined the statistical interpretation was lost in translation, leaving behind a false sense of conclusiveness.
Dr Charles Shepherd
Honorary Medical Adviser, ME Association
This preliminary reanalysis of the PACE trial data fully vindicates the deep concerns and scepticism that have been expressed by both doctors and people with ME/CFS about the efficacy and recovery rates following the use of CBT and graded exercise therapy in this controversial clinical trial.
The reanalysis also indicates that the decision to weaken the recovery thresholds in the original protocol, after the trial had started, has significantly inflated the effectiveness of CBT and graded exercise therapy. It was the scientific equivalent of moving the winning post after the race had started.
As a result, NICE must urgently reconsider their guidance that CBT and graded exercise therapy should be recommended as primary forms of treatment for everyone with mild or moderate ME/CFS. Other organisations making similar recommendations, such as the CDC in America, must also reconsider their guidance on CBT and graded exercise therapy.
The editor of The Lancet must also study this disturbing reanalysis of the recovery rate results. And if it is confirmed by a more detailed and complete analysis of the data that the PACE trial conclusions are unsound, then the PACE trial papers must be retracted
Zaher Nahle, PhD, MPA
Vice President for Research and Scientific Programs, Solve ME/CFS Initiative
This preliminary reanalysis of the PACE trial outcomes obtained from patient data acquired through an FOIA request is a huge step forward, a watershed moment that has been awaited with anticipation by our community for far too long. It is also jaw-dropping in what it uncovered.
While it confirmed what patients knew all along regarding the fallacies peddling the benefit of CBT and graded exercise therapy in ME/CFS, it exposed, using the trial's empirical data, disturbing facts related to the findings and outcomes of that trial, and sloppy science.
Two immediate steps must now follow: 1) a retraction of the original paper with an immediate public erratum detailing this new discovery. While the retraction process is in motion, the opportunity should be given to the study authors to examine and accept this new re-analysis; 2) a moratorium on all ongoing research activities including grants, trials, proceedings or any scholarly activities that were based in full or in part on the original trial results and its now challenged conclusions.
Dr Simon John Duffy
Director, Centre for Welfare Reform
The scandalous distortion of academic research within the PACE research trial is now even clearer. Proper analysis of the data demonstrates what commentators had feared — that there is no significant evidence as to the effectiveness of CBT and graded exercise as therapies to help people recover from chronic fatigue syndrome.
PACE was the first medical trial to receive funding from the Department of Work and Pensions, and was expected to validate an approach to disability that has been used to justify their dubious ‘welfare reform’ agenda. Results were then presented in a way that created a misleading impression of success.
It seems that many in the UK's research establishment have failed to question or challenge this flawed research, despite growing international criticism. The losers have been disabled people, and in particular those people with ME or chronic fatigue syndrome. Independent academic research is vital, but it is clear that the current cocktail of research funding by Government and commerce, combined with ideological prejudice, has opened the door to unjustifiable assertions and bad public policy.
Carly Maryhew, JD
Patient and co-author of reanalysis paper
As shocking as the actual "recovery" numbers are, it's even more outrageous that patients had to go to a tribunal to have their concerns heard. The flaws in the methodology of the PACE trial were obvious as soon as the original paper was published in The Lancet, yet researchers and their universities strongly resisted any attempt to make the anonymous data from a publicly funded trial publicly available. We need to look beyond what went wrong with the data in the PACE trial, and look at what went wrong with the British academic research community, to prevent this sort of fiasco from happening again.
Information Officer, Irish ME/CFS Association
The revised thresholds for recovery should never have been accepted for publication. All were flawed and in two cases there was the ridiculous scenario where you could deteriorate on the measure and be counted as recovered. How could this have happened in a large, publicly funded trial that was supposed to give "definitive" answers on the therapies in question? One is left wondering whether prejudice about chronic fatigue syndrome and willingness to treat those who suffer from it as third-class citizens led to lower standards.
It is not in the interests of healthcare professionals nor patients to have misleading claims about recovery and, in particular, a false expectation that patients with chronic fatigue syndrome can recover with cognitive behaviour therapy or a graded exercise regime.
More generally, this case will hopefully serve as an example of the importance of open data and not allowing investigators with strong allegiances to control how medical trial data is reported, especially when late changes have been made to the analysis method.
Patient and science blogger
Patients are desperate to recover from this life-wrecking illness and want to know their chances of recovery with treatment. The 22% recovery figure after CBT or graded exercise sounds worthwhile. I'd want some of that. But it turns out that the real figure — the one the researchers had promised to calculate before starting the trial — is way, way lower and that it's essentially no different to what you'd get without the therapies.
This was a publicly funded trial, run to find out if these treatments really work, and patients need reliable information about effectiveness. The researchers shouldn't be watering down their definition of recovery in a way that makes their results look better than they really are.
How can there be evidence-based guidelines for treatment when the evidence is unreliable?
Executive Director, The Young ME Sufferers Trust (Tymes Trust)
How many people with ME have been damaged by this exercise dogma? You only need to listen to parents whose children have been forced through it to know what happens. The NICE guideline should immediately be changed.
- End –
Paper describing the new analyses by Matthees and colleagues (attached to Virology Blog article on the implications, by Dr David Tuller): http://www.virology.ws/2016/09/21/no-recovery-in-pace-trial-new-analysis-finds/
PACE authors’ 2013 paper in Psychological Medicine on recovery rates: https://www.cambridge.org/core/journals/psychological-medicine/article/recovery-from-chronic-fatigue-syndrome-after-treatments-given-in-the-pace-trial/6E3A5B37D427239D146C5EB276A11E06
PACE authors’ 2011 paper in The Lancet: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096-2/fulltext
What is ME (Action for ME): https://www.actionforme.org.uk/what-is-me/introduction/
Tribunal orders release of PACE data (#MEAction): http://www.meaction.net/2016/08/16/tribunal-orders-release-of-pace-data/
PACE research sparked patient rebellion challenged medicine (STATS.org): http://www.stats.org/pace-research-sparked-patient-rebellion-challenged-medicine/
Editorial on PACE (STATS.org): http://www.stats.org/editorial-on-pace/
ME/CFS articles by Dr David Tuller and others (Virology Blog): http://www.virology.ws/mecfs/
#MEAction is an international network of patients empowering each other to fight for health equality for Myalgic Encephalomyelitis. #MEAction is not structured like a traditional advocacy organization. We are a platform designed to empower patients advocates and organizations, wherever they might be, with the technological tools and training to do what they are already doing – better. We aim to become a mass, grassroots organization by 2017. For more information, visit http://www.meaction.net/.