Why the scientific community needs the PACE trial data to be released

To_deposit_or_not_to_deposit,_that_is_the_question_-_journal.pbio.1001779.g001University and clinical trial investigators must release data to a citizen-scientist patient, according to a landmark decision in the UK. But the decision could still be overturned if the University and investigators appeal. The scientific community needs the decision to be upheld. I’ll argue that it’s unwise for any appeal to be made. The reasons for withholding the data in the first place were archaic. Overturning of the decision would set a bad precedent and would remove another tooth from almost toothless requirements for data sharing.

We didn’t need Francis Collins, Director of National Institutes of Health to tell us what we already knew, the scientific and biomedical literature is untrustworthy.

And there is the new report from the UK Academy of Medical Sciences, Reproducibility and reliability of biomedical research: improving research practice.

There has been a growing unease about the reproducibility of much biomedical research, with failures to replicate findings noted in high-profile scientific journals, as well as in the general and scientific media. Lack of reproducibility hinders scientific progress and translation, and threatens the reputation of biomedical science.

Among the report’s recommendations:

  • Journals mandating that the data underlying findings are made available in a timely manner. This is already required by certain publishers such as the Public Library of Science (PLOS) and it was agreed by many participants that it should become more common practice.
  • Funders requiring that data be released in a timely fashion. Many funding agencies require that data generated with their funding be made available to the scientific community in a timely and responsible manner

A consensus has been reached: The crisis in the trustworthiness of science can be only overcome only if scientific data are routinely available for reanalysis. Independent replication of socially significant findings is often unfeasible, and unnecessary if original data are fully available for inspection.

Numerous governmental funding agencies and regulatory bodies are endorsing routine data sharing.

The UK Medical Research Council (MRC) 2011 policy on data sharing and preservation  has endorsed principles laid out by the Research Councils UK including

Publicly funded research data are a public good, produced in the public interest, which should be made openly available with as few restrictions as possible in a timely and responsible manner.

To enable research data to be discoverable and effectively re-used by others, sufficient metadata should be recorded and made openly available to enable other researchers to understand the research and re-use potential of the data. Published results should always include information on how to access the supporting data.

The Wellcome Trust Policy On Data Management and Sharing opens with

The Wellcome Trust is committed to ensuring that the outputs of the research it funds, including research data, are managed and used in ways that maximise public benefit. Making research data widely available to the research community in a timely and responsible manner ensures that these data can be verified, built upon and used to advance knowledge and its application to generate improvements in health.

The Cochrane Collaboration has weighed in that there should be ready access to all clinical trial data

Summary results for all protocol-specified outcomes, with analyses based on all participants, to become publicly available free of charge and in easily accessible electronic formats within 12 months after completion of planned collection of trial data;

Raw, anonymised, individual participant data to be made available free of charge; with appropriate safeguards to ensure ethical and scientific integrity and standards, and to protect participant privacy (for example through a central repository, and accompanied by suitably detailed explanation).

Many similar statements can be found on the web. I’m unaware of credible counterarguments gaining wide acceptance.

toothless manYet, endorsements of routine sharing of data are only a promissory reform and depend on enforcement that has been spotty, at best. Those of us who request data from previously published clinical trials quickly realize that requirements for sharing data have no teeth. In light of that, scientists need to watch closely whether a landmark decision concerning sharing of data from a publicly funded trial is appealed and overturned.

The Decision requiring release of the PACE data

The UK’s Information Commissioner’s Office (ICO) ordered Queen Mary University of London (QMUL) on October 27, 2015 to release anonymized from the PACE chronic fatigue syndrome trial data to an unnamed complainant. QMUL has 28 days to appeal.

Even if scientists don’t know enough to care about Chronic Fatigue Syndrome/Myalgic Encephalomyelitis, they should be concerned about the reasons that were given in a previous refusal to release the data.

I took a critical look at the long-term follow up results for the PACE trial in a previous Mind the Brain blog post  and found fatal flaws in the authors’ self-congratulatory interpretation of results. Despite authors’ claims to the contrary and their extraordinary efforts to encourage patients to report the intervention was helpful, there were simply no differences between groups at follow-up

Background on the request for release of PACE data

  • A complainant requested release of specific PACE data from QMUL under the Freedom of Information Act.
  • QMUL refused the request.
  • The complainant requested an internal review but QMUL maintained its decision to withhold the data.
  • The complainant contacted the ICO with concerns about how the request had been handled.
  • On October 27, 2015, the ICO sided with the complainant and order the release of the data.

A report outlines Queen Mary’s arguments for refusing to release the data and the Commissioner’s justification for siding with the patient requesting the data be released.

Reasons the request release of data was initially refused

The QMU PACE investigators claimed

  • They were entitled to withhold data prior to publication of planned papers.
  • An exemption to having to share data because data contained sensitive medical information from which it was possible to identify the trial participants.
  • Release of the data might harm their ability to recruit patients for research studies in the future.

The QMU PACE researchers specifically raised concerns about a motivated intruder being able to facilitate re-identification of participants:

In relation to a motivated intruder being able facilitate re-identification of participants, the University argued that:

“The PACE trial has been subject to extreme scrutiny and opponents have been against it for several years. There has been a concerted effort by a vocal minority whose views as to the causes and treatment of CFS/ME do not comport with the PACE trial and who, it is QMUL’s belief, are trying to discredit the trial. Indeed, as noted by the editor of the Lancet, after the 2011 paper’s publication, the nature of this comprised not a ‘scientific debate’ but an “orchestrated response trying to undermine the credibility of the study from patient groups [and]… also the credibility of the investigators and that’s what I think is one of the other alarming aspects of this. This isn’t a purely scientific debate; this is going to the heart of the integrity of the scientists who conducted this study.”

Magneto_430Bizarre. This is obviously a talented masked motivated intruder. Do they have evidence that Magneto is at it again? Mostly he now is working with the good guys, as seen in the help he gave Neurocritic and me.

Let’s think about this novel argument. I checked with University of Pennsylvania bioethicist Jon Merz, an expert who has worked internationally to train researchers and establish committees for the protection of human subjects. His opinion was clear:

The litany of excuses – not reasons – offered by the researchers and Queen Mary University is a bald attempt to avoid transparency and accountability, hiding behind legal walls instead of meeting their critics on a level playing field.  They should be willing to provide the data for independent analyses in pursuit of the truth.  They of course could do this willingly, in a way that would let them contractually ensure that data would be protected and that no attempts to identify individual subjects would be made (and it is completely unclear why anyone would care to undertake such an effort), or they can lose this case and essentially lose any hope for controlling distribution.

The ‘orchestrated response to undermine the credibility of the study’ claimed by QMU and the PACE investigators, as well as issue being raised of the “integrity of the scientists who conducted the study” sounds all too familiar. It’s the kind of defense that is heard from scientists under scrutiny of the likes of Open Science Collaborations, as in psychology and cancer. Reactionaries resisting post-publication peer review say we must be worried about harassment from

“replication police” “shameless little bullies,” “self-righteous, self-appointed sheriffs” engaged in a process “clearly not designed to find truth,” “second stringers” who were incapable of making novel contributions of their own to the literature, and—most succinctly—“assholes.”

Far fetched? Compare this to a QMU quote drawn from the National Radio, Australian Broadcast Company April 18, 2011 interview of Richard Horton and PACE investigator Michael Sharpe in which former Lancet Editor Richard Horton condemned:

A fairly small, but highly organised, very vocal and very damaging group of individuals who have…hijacked this agenda and distorted the debate…

dost thou feel‘Distorted the debate’? Was someone so impertinent as to challenge investigators’ claims about their findings? Sounds like Pubpeer  We have seen what they can do.

Alas, all scientific findings should be scrutinized, all data relevant to the claims that are made should be available for reanalysis. Investigators just need to live with the possibility that their claims will be proven wrong or exaggerated. This is all the more true for claims that have substantial impact on public policy and clinical services, and ultimately, patient welfare.

[It is fascinating to note that Richard Horton spoke at the meeting that produced the UK Academy of Medical Sciences report to which I provided a link above. Horton covered the meaning in a Lancet editorial  in which he amplified the sentiment of the meeting: “The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world.” His editorial echoed a number of recommendations of the meeting report, but curiously omitted mentioning of data sharing.]

jacob-bronowski-scientist-that-is-the-essence-of-science-ask-anFortunately the ICO has rejected the arguments of QMUL and the PACE investigators. The Commissioner found that QMUL and the PACE investigators incorrectly interpreted regulations in their withholding of the data and should provide the complaint with the data or risk being viewed as in contempt of court.

The 30-page decision is a fascinating read, but here’s an accurate summary from elsewhere:

In his decision, the Commissioner found that QMUL failed to provide any plausible mechanism through which patients could be identified, even in the case of a “motivated intruder.” He was also not convinced that there is sufficient evidence to determine that releasing the data would result in the mass exodus of a significant number of the trial’s 640 participants nor that it would deter significant numbers of participants from volunteering to take part in future research.

Requirements for data sharing in the United States have no teeth and situation would be worsened by reversal of ICO decision

Like the UK, the United States supposedly has requirements for sharing of data from publicly funded trials. But good luck in getting support from regulatory agencies associated with funding sources for obtaining data. Here’s my recent story, still unfolding – or maybe, sadly, over, at least for now.

For a long time I’ve fought my own battles about researchers making unwarranted claims that psychotherapy extend the lives of cancer patients. Research simply does not support the claim. The belief that psychological factors have such influence on the course and outcome of cancer sets up cancer patients to be blamed and to blame themselves when they don’t overcome their disease by some sort of mind control. Our systematic review concluded

“No randomized trial designed with survival as a primary endpoint and in which psychotherapy was not confounded with medical care has yielded a positive effect.”

Investigators who conducted some of the best ambitious, well-designed trials to test the efficacy of psychological interventions on cancer but obtained null results echoed our assessment. The commentaries were entitled “Letting Go of Hope” and “Time to Move on.”

I provided an extensive review of the literature concerning whether psychotherapy and support groups increased survival time in an earlier blog post. Hasn’t the issue of mind-over-cancer been laid to rest? I was recently contacted by a science journalist interested in writing an article about this controversy. After a long discussion, he concluded that the issue was settled — no effect had been found — and he could not succeed in pitching his idea for an article to a quality magazine.

But as detailed here one investigator has persisted in claims that a combination of relaxation exercises, stress reduction, and nutritional counseling increases survival time. My colleagues and I gave this 2008 study a careful look.  We ran chi-square analyses of basic data presented in the paper’s tables. But none of our analyses of group assignment on mortality more disease recurrence was significant. The investigators’ claim of an effect depended on dubious multivariate analyses with covariates that could not be independently evaluated without a look at the data.

The investigator group initially attempted to block publication of a letter to the editor, citing a policy of the journal Cancer that critical letters could not be published unless investigators agreed to respond and they were refusing to respond. We appealed and the journal changed its policy and allowed us additional length to our letter.

We then requested from the investigator’s University Research Integrity Officer the specific data needed to replicate the multivariate analyses in which the investigators claimed an effect on survival. The request was denied:

The data, if disclosed, would reveal pending research ideas and techniques. Consequently, the release of such information would put those using such data for research purposes in a substantial competitive disadvantage as competitors and researchers would have access to the unpublished intellectual property of the University and its faculty and students.

Recall that we were requesting in 2014 specific data needed to evaluate analyses published in 2008.

I checked with statistician Andrew Gelman whether my objections to the multivariate analyses were well-founded and he agreed they were.

Since then, another eminent statistician Helena Kraemer has published an incisive critique of reliance in a randomized controlled trial on multivariate analyses and simple bivariate analyses do not support the efficacy of interventions. She labeled adjustments with covariates as a “source of false-positive findings.”

We appealed to the US Health and Human Services Office of Research Integrity  (ORI) but they indicated no ability to enforce data sharing.

Meanwhile, the principal investigator who claimed an effect on survival accompanied National Cancer Institute program officers to conferences in Europe and the United States where she promoted her intervention as effective. I complained to Robert Croyle, Director, NCI Division of Cancer Control and Population Sciences who twice has been one of the program officer’s co-presenting with her. Ironically, in his capacity as director he is supposedly facilitating data sharing for the division. Professionals were being misled to believe that this intervention would extend the lives of cancer patients, and the claim seemingly had the endorsement NCI.

I told Robert Croyle  that if only the data for the specific analyses were released, it could be demonstrated that the claims were false. Croyle did not disagree, but indicated that there was no way to compel release of the data.

The National Cancer Institute recently offered to pay the conference fees to the International Psycho-Oncology Congress in Washington DC of any professionals willing to sign up for free training in this intervention.

I don’t think I could get any qualified professional including  Croyle to debate me publicly as to whether psychotherapy increases the survival of cancer patients. Yet the promotion of the idea persists because it is consistent with the power of mind over body and disease, an attractive talking point

I have not given up in my efforts to get the data to demonstrate that this trial did not show that psychotherapy extends the survival of cancer patients, but I am blocked by the unwillingness of authorities to enforce data sharing rules that they espouse.

There are obvious parallels between the politics behind persistence of the claim in the US for psychotherapy increasing survival time for cancer patients and those in the UK about cognitive behavior therapy being sufficient treatment for schizophrenia in the absence of medication or producing recovery from the debilitating medical condition, Chronic Fatigue Syndrome/Myalgic Encephalomyelitis. There are also parallels to investigators making controversial claims based on multivariate analyses, but not allowing access to data to independently evaluate the analyses. In both cases, patient well-being suffers.

If the ICO upholds the release of data for the PACE trial in the UK, it will pressure the US NIH to stop hypocritically endorsing data sharing and rewarding investigators whose credibility depends on not sharing their data.

As seen in a PLOS One study, unwillingness to share data in response to formal requests is

associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance.

Why the PACE investigators should not appeal

In the past, PACE investigators have been quite dismissive of criticism, appearing to have assumed that being afflicted with Chronic Fatigue Syndrome/Myalgic Encephalomyelitis precludes a critic being taken seriously, even when the criticism is otherwise valid. However, with publication of the long-term follow-up data in Lancet Psychiatry, they are now contending with accomplished academics whose criticisms cannot be so easily brushed aside. Yes, the credibility of the investigators’ interpretations of their data are being challenged. And even if they do not believe they need to be responsive to patients, they need to be responsive to colleagues. Releasing the data is the only acceptable response and not doing so risks damage to their reputations.

QMUL, Professors White and Sharpe, let the People’s data go.


35 thoughts on “Why the scientific community needs the PACE trial data to be released”

  1. Thank you very much indeed for this new post. I hope the PACE authors and Queen Mary University of London don’t appeal the ICO’s decision but their past attitude gives me no faith in them.

    A petition that includes a call to the study authors to “publish the recovery outcomes according to the analyses specified in the trial’s protocol and to give independent researchers full access to the raw data (anonymised by removing trial identifiers and all other data superfluous to the calculation, such as age, sex or location)” has already gained over 9,300 signatures in less than two weeks.

    The petition host, #MEAction, has even undertaken to meet any reasonable cost of analysis or data preparation.

    I hope all your readers will sign the petition. We have nearly reached our 10,000 target.



  2. Thanks again for a wonderful article 🙂 The hypocrisy of these institutions in publicly endorsing data-sharing, while consistently blocking access to such data, is shocking and should not be permitted to continue. If those institutions wish to protect their researchers when they engage in dubious methodology, they should at least have the decency to be honest about their refusal to share such data.


  3. Thank you Dr Coyne.

    Would you be interested in getting involved in analysing it? I don’t have any connection to the individual that has made the request. This is just to know whether you would be interested.


  4. ME/CFS patients are grateful for Prof. Coyne and other researchers, journalists and advocates who are coming to our aid in fighting bad trials and the psychiatric hold on a biologic disease. Any chronic illness will from time to time need the support of a psychiatrist (MS, ALS, Diabetes, Lupus, Lyme, etc.) but none of these are mental health issues.

    Psychiatry has enough valid mental health issues to work on and many individuals are suffering from clinical depression, PTSD, GAD, etc., that need their attention and financial resources. ME/CFS is not one of them and bio research by bio researchers must be done and the NIH has committed itself to that research.

    The PACE trial must be retracted from the Lancet. The UK Psych Lobby must no longer control the message of the ME/CFS patients and community.


  5. The 2011 Research Councils UK (RCUK) policy states that: “Publicly funded research data are a public good, produced in the public interest, which should be made openly available with as few restrictions as possible in a timely and responsible manner that does not harm intellectual property”.

    RCUK requires all funded universities to have a data management policy and road map that will meet their
    expectations for data sharing in place by 1 May 2012 with full implementation by 2015. One requirement is
    that “publicly funded research data should be made openly available in a timely manner”.

    This information is from QMUL’s website:

    I wonder if RCUK is aware of this story?

    In any case, 2015 isn’t over yet, and QMUL can still make the data available 😉


  6. If funding bodies refused new grants to institutions that failed to share data it would be quickly shared. The other approach would be to mandate third party collection and release of the anonymised data on a specific date (say 1 year after last sample recorded).


  7. The sort of data wanted would be rows of entries such as 60, 65, 28, 25, 10, 10, yes, yes, 4,3, 321, 332, CBT. How on earth would anyone track down who that was? Is this the sensitive medical data that, according to QMUL, would enable people to identify the participants? It is simply a knee-jerk reaction against patients: a supposition that having ME suddenly wipes out any scientific or analytic skills that we have, and a fear of what we may find.


  8. Thank you once again, Prof Coyne for your timely and erudite contribution to this debate.

    To my mind, the fundamental issue here is of QMUL’s deliberate intent to ensure that no-one else is able to undertake a proper investigation of the PACE trial. As noted by Popper and many others, an integral principle of the scientific method is that research results must be reproducible. Thus, all relevant data, code and calculations must be publicly released. Denying access to the data to others is not only unethical but also anti-science.

    The actions of the PACE authors and others (eg, “it was found that 246 out of 394 contacted authors of papers in APA journals did not share their data upon request”1) demonstrate that science is in crisis. Perhaps all scientists should be sent back to school to re-learn the Scientific Method?

    1. Vanpaemel, W.; Vermorgen, M.; Deriemaecker, L.; Storms, G. (2015). “Are we wasting a good crisis? The availability of psychological research data after the storm”. Collabra 1 (1): 1–5.


  9. Why the scientific community needs the PACE trial data to be released

    The release of PACE trial data is in the best interest of the public for countless reasons.

    If the data is good and a comprehensive subsequent analysis by reputable research statisticians confirms or refutes earlier claims made, then TRUTH has prevailed, regardless of claims by skeptics to the contrary.

    Whatever the outcome is, it is for the benefit of all of the sciences. Science’s reputation will thus be upheld as the most reliable source of knowledge we can thus far in human history continue to use for growth and remediation.

    Hopefully, Queen Mary University of London (QMUL) will lean in this direction and recognise the greater good involved and do the right thing.


  10. Thank you for this excellent post!

    I agree with you completely, except let me point out that serious academics such as Prof. Em. Malcolm Hooper and Angela Kennedy have been harshly (and appropriately) criticising this $8M publicly funded study and repeatedly calling for the release of data since the start. It has not just been patient-experts calling for this from the beginning.

    I look very forward to reading more on this from you!


  11. I never know whether to laugh or cry when I see highly qualified researchers and journal editors claiming that people criticising the science of the PACE papers are “highly organised” and that the response is orchestrated.

    They are so caught up in a magnificent conspiracy theory of a clandestine gang of moustache-twirling villains trying to maliciously derail their heroic efforts. But the actual simple truth is that there are a lot of people with ME who are still just about well enough to take an interest in keeping up with scientific journals, and when they read bad science (particularly on a subject that affects them on a personal level) their spontaneous emotional reaction is to want to try to correct the science and address the flaws they see. That’s what happens in the scientific process – you test your theory, and then someone comes along and says “But what about [x]?” and you go back and test for [x] or you say “Ah. Oops.” and the science gets better because of the interaction.

    Many people with ME have scientific degrees, and many have medical degrees, and many friends, family and carers of ME patients are similarly well-qualified to assess the research. Not to mention, there are many other ME researchers both in the UK and across the world who disagree with the science and the results in the PACE trial…but somehow, when it’s being talked about in the press, when the conspiracy theory rears it’s ugly head again (and again and again!) the critical researchers are never mentioned; it’s only the patients who are organised and reactionary and somehow colluding to deny reality.


  12. Thank you for another excellent analysis of the situation. I was particularly intrigued by the cancer research, as I have seen this used as an ‘argument’ in favour of CBT as a treatment for ME/CFS, e.g. that this somehow proves that using CBT for ME/CFS doesn’t mean it’s a psychological illness, when in fact the objections are that CBT is not used as a frontline treatment for cancer as it is recommended by the British NICE guideline for doctors, and that the CBT protocol for ME/CFS as per PACE trial is not the same as for other chronic illnesses, but is based on the premise of absence of underlying organic disease. Thank you again, I really appreciate your objectivism on this serious issue for global public health.


  13. What is it with our culture’s incessant need to push the idea that thoughts influence hard outcomes? I don’t find it much different from primitive cultures and their assigning disease to bad spirits and curses.


  14. The Pubmed entry http://www.ncbi.nlm.nih.gov/pubmed/21334061 of the paper lists 20 authors:
    B J Angus, H L Baber, J Bavinton, M Burgess, T Chalder, L V Clark, D L Cox, J C DeCesare, K A Goldsmith, A L Johnson, P McCrone, G Murphy, M Murphy, H O’Dowd, PACE trial management group, L Potts, M Sharpe, R Walwyn, D Wilks and P D White (re-arranged in an alfabetic order).
    Another entry ( http://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736(11)60096-2.pdf and also http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096-2/fulltext ) lists 19 authors. Both state that all 19 authors are acting ‘on behalf of the PACE trial management group†’. and “†Members listed at end of paper”. The end of the paper (page 835) states: “PACE trial group.” This term is not identical to “PACE trial management group”.
    In total another 19 names are listed under “PACE trial group” (page 835): Hiroko Akagi, Mansel Aylward, Barbara Bowman Jenny Butler, Chris Clark, Janet Darbyshire, Paul Dieppe, Patrick Doherty, Charlotte Feinmann, Deborah Fleetwood, Astrid Fletcher, Stella Law, M Llewelyn, Alastair Miller, Tom Sensky, Peter Spencer, Gavin Spickett, Stephen Stansfeld and Alison Wearden (re-arranged in an alfabetic order).
    There is no overlap with the first 19 people who are listed as author of the paper.
    So how many people can claim to be an author of this paper? Are all these 19 people of the “PACE trial management group” (identical to “PACE trial group”???) also some sort of co-author of this paper? Do all these 19 people of the second group also agree with the complete contents of the paper? Do all 38 people agree with the full contents of the paper?
    The paper lists many affiliations:
    * Queen Mary University of London, UK
    * King’s College London, UK
    * University of Cambridge, UK
    * University of Cumbria, UK
    * University of Oxford, UK
    * University of Edinburgh, UK
    * Medical Research Council Clinical Trials Unit, London, UK
    * South London and Maudsley NHS Foundation Trust, London, UK
    * The John Radcliffe Hospital, Oxford, UK
    * Royal Free Hospital NHS Trust, London, UK
    * Barts and the London NHS Trust, London, UK
    * Frenchay Hospital NHS Trust, Bristol, UK;
    * Western General Hospital, Edinburgh, UK
    Do all these affilations also agree with the full contents of the paper? Am I right to assume that all 38 people (names see above) and all affilations / institutes (see above) plainly refuse to give critics / other scientists / patients / patient groups (etc.) access to the raw research data of this paper and am I am right with my assumption that it is therefore impossible for all others (including allies of patients / other scientists / interested students, etc.) to conduct re-calculations, check all statements with the raw data, etc?
    Excuse me very much, but I totally fail to understand how this is possible, (1) in the Dutch context, (2) in the context of the policy of the funders, (3) in the context of the public interest in this topic, (4) in the context of how to learn students to become a good researcher, etc.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s