School-Based Mindfulness Based Stress-Reduction Program (MBSR) fails to deliver positive results

No positive effects found for Jon Kabat-Zinn’s Mindfulness Based Stress-Reduction Program with middle and high school students. Evidence of deterioration was found in some subgroup analyses.

mind the brain logo

No positive effects found for Jon Kabat-Zinn’s Mindfulness Based Stress-Reduction Program with middle and high School Students. Evidence of deterioration was found in some subgroup analyses.

mindfulness in schoolsWe should be cautious about interpreting negative effects that are confined to subgroup analyses. They may well be due to chance. But we should be concerned about the lack of positive findings across measures in the primary analyses. MBSR (a mindfulness training product trademarked and controlled by Jon Kabat-Zinn) and other mindfulness programs have heavily promoted as having wondrous benefits and mandated in many school settings.

The study [with link to the PDF]

Johnson C, Burke C, Brinkman S, Wade T. Effectiveness of a school-based mindfulness program for transdiagnostic prevention in young adolescents. Behaviour Research and Therapy. 2016 Jun 30;81:1-1.

Abstract

Anxiety, depression and eating disorders show peak emergence during adolescence and share common risk factors. School-based prevention programs provide a unique opportunity to access a broad spectrum of the population during a key developmental window, but to date, no program targets all three conditions concurrently. Mindfulness has shown promising early results across each of these psychopathologies in a small number of controlled trials in schools, and therefore this study investigated its use in a randomised controlled design targeting anxiety, depression and eating disorder risk factors together for the first time. Students (M age 13.63; SD = .43) from a broad band of socioeconomic demographics received the eight lesson, once weekly.b (“Dot be”) mindfulness in schools curriculum (N = 132) or normal lessons (N = 176). Anxiety, depression, weight/shape concerns and wellbeing were the primary outcome factors. Although acceptability measures were high, no significant improvements were found on any outcome at post-intervention or 3-month follow-up. Adjusted mean differences between groups at post-intervention were .03 (95% CI: -.06 to -.11) for depression, .01 (-.07 to -.09) for anxiety, .02 (-.05 to -.08) for weight/shape concerns, and .06 (-.08 to -.21) for wellbeing. Anxiety was higher in the mindfulness than the control group at follow-up for males, and those of both genders with low baseline levels of weight/shape concerns or depression. Factors that may be important to address for effective dissemination of mindfulness-based interventions in schools are discussed. Further research is required to identify active ingredients and optimal dose in mindfulness-based interventions in school settings.

The discussion noted:

The design of this study addresses several shortcomings identified in the literature (Britton et al., 2014; Burke, 2010; Felver et al., 2015; Meiklejohn et al., 2012; Tan, 2015; Waters et al., 2014). First, it was a multi-site, randomised controlled design with a moderately large sample size based on a priori power calculations. Second, it included follow-up (three months). Third, it sought to replicate an existing mindfulness-based intervention for youth. Fourth, socioeconomic status was not only reported but a broad range of socioeconomic bands included, although it was unfortunate that poor opt-in consent rates resulted in high data wastage in the lower range schools. Use of the same instructor for all classes in the intervention arm represents a strength (consistency) and a limitation (generalisability of findings).

Coverage in Scientific American

Mindfulness Training for Teens Fails Important Test

A large trial in schools showed no evidence of benefits, and hints it could even cause problems

The fact that this carefully-controlled investigation showed no benefits of mindfulness for any measure, and furthermore indicated an adverse effect for some participants, indicates that mindfulness training is not a universal solution for addressing anxiety or depression in teens, nor does it qualify as a replacement for more traditional psychotherapy or psychopharmacology, at least not as implemented in this school-based paradigm.

eBook_Mindfulness_345x550Preorders are being accepted for e-books providing skeptical looks at mindfulness and positive psychology, and arming citizen scientists with critical thinking skills. Right now there is a special offer for free access to a Mindfulness Master Class. But hurry, it won’t last.

I will also be offering scientific writing courses on the web as I have been doing face-to-face for almost a decade. I want to give researchers the tools to get into the journals where their work will get the attention it deserves.

Sign up at my website to get advance notice of the forthcoming e-books and web courses, as well as upcoming blog posts at this and other blog sites. Get advance notice of forthcoming e-books and web courses. Lots to see at CoyneoftheRealm.com.
 

“It’s certainly not bareknuckle:” Comments to a journalist about a critique of mindfulness research

We can’t assume authors of mindfulness studies are striving to do the best possible science, including being prepared for the possibility of being proven incorrect by their results.

mind the brain logo

I recently had a Skype interview with science journalist Peter Hess concerning an article in Psychological Science.

Peter was exceptionally prepared, had a definite point of view, but was open to what I said. In the end seem to be persuaded by me on a number of points.  The resulting article in Inverse  faithfully conveyed my perspective and juxtaposed quotes from me with those from an author of the Psych Science piece in a kind of debate.

My point of view

larger dogWhen evaluating an article about mindfulness in a peer-reviewed journal, we need to take into account that authors may not necessarily be striving to do the best science, but to maximally benefit their particular brand of mindfulness, their products, or the settings in which they operate. Many studies of mindfulness are a little more than infomercials, weak research intended only to get mindfulness promoters’ advertisement of themselves into print or to allow the labeling of claims as “peer-reviewed”. Caveat Lector.

We cannot assume authors of mindfulness studies are striving to do the best possible science, including being prepared for the possibility of being proven incorrect by their results. Rather they may be simply try to get the strongest possible claims through peer review, ignoring best research practices and best publication practices.

Psychologists Express Growing Concern With Mindfulness Meditation

“It’s not bare-knuckle, that’s for sure.”

There was much from the author of the Psych Science article with which  I would agree:

“In my opinion, there are far too many organizations, companies, and therapists moving forward with the implementation of ‘mindfulness-based’ treatments, apps, et cetera before the research can actually tell us whether it actually works, and what the risk-reward ratio is,” corresponding author and University of Melbourne research fellow Nicholas Van Dam, Ph.D. tells Inverse.

Bravo! And

“People are spending a lot of money and time learning to meditate, listening to guest speakers about corporate integration of mindfulness, and watching TED talks about how mindfulness is going to supercharge their brain and help them live longer. Best case scenario, some of the advertising is true. Worst case scenario: very little to none of the advertising is true and people may actually get hurt (e.g., experience serious adverse effects).”

But there were some statements that renewed the discomfort and disappointment I experienced when I read the original article in Psychological Science:

 “I think the biggest concern among my co-authors and I is that people will give up on mindfulness and/or meditation because they try it and it doesn’t work as promised,” says Van Dam.

“There may really be something to mindfulness, but it will be hard for us to find out if everyone gives up before we’ve even started to explore its best potential uses.”

So, how long before we “give up” on thousands of studies pouring out of an industry? In the meantime, should consumers act on what seem to be extravagant claims?

The Inverse article segued into some quotes from me after delivering another statement from the author which I could agree:

The authors of the study make their attitudes clear when it comes to the current state of the mindfulness industry: “Misinformation and poor methodology associated with past studies of mindfulness may lead public consumers to be harmed, misled, and disappointed,” they write. And while this comes off as unequivocal, some think they don’t go far enough in calling out specific instances of quackery.

“It’s not bare-knuckle, that’s for sure. I’m sure it got watered down in the review process,” James Coyne, Ph.D., an outspoken psychologist who’s extensively criticized the mindfulness industry, tells Inverse.

Coyne agrees with the conceptual issues outlined in the paper, specifically the fact that many mindfulness therapies are based on science that doesn’t really prove their efficacy, as well as the fact that researchers with copyrights on mindfulness therapies have financial conflicts of interest that could influence their research. But he thinks the authors are too concerned with tone policing.

“I do appreciate that they acknowledged other views, but they kept out anybody who would have challenged their perspective,” he says.

Regarding Coyne’s criticism about calling out individuals, Van Dam says the authors avoided doing that so as not to alienate people and stifle dialogue.

“I honestly don’t think that my providing a list of ‘quacks’ would stop people from listening to them,” says Van Dam. “Moreover, I suspect my doing so would damage the possibility of having a real conversation with them and the people that have been charmed by them.” If you need any evidence of this, look at David “Avocado” Wolfe, whose notoriety as a quack seems to make him even more popular as a victim of “the establishment.” So yes, this paper may not go so far as some would like, but it is a first step toward drawing attention to the often flawed science underlying mindfulness therapies.

To whom is the dialogue directed about unwarranted claims from the mindfulness industry?

As one of the authors of an article claiming to be an authoritative review from a group of psychologists with diverse expertise, Van Dam says he is speaking to consumers. Why won’t he and his co-authors provide citations and name names so that readers can evaluate for themselves what they are being told? Is the risk of reputational damage and embarrassment to the psychologists so great as to cause Van Dam to protect them versus protecting consumers from the exaggerated and even fraudulent claims of psychologists hawking their products branded as ‘peer-reviewed psychological and brain science’.

I use the term ‘quack’ sparingly outside of discussing unproven and unlikely-to-be-proven products supposed to promote physical health and well-being or to prevent or cure disease and distress.

I think Harvard psychologist Ellen Langer deserves the term “quack” for her selling of expensive trips to spas in Mexico to women with advanced cancer so that they can change their mind set to reverse the course of their disease. Strong evidence, please! Given that this self-proclaimed mother of mindfulness gets her claims promoted through the Association for Psychological Science website, I think it particularly appropriate for Van Dam and his coauthors to name her in their publication in an APS journal. Were they censored or only censoring themselves?

Let’s put aside psychologists who can be readily named as quacks. How about Van Dam and co-authors naming names of psychologists claiming to alter the brains and immune systems of cancer patients with mindfulness practices so that they improve their physical health and fight cancer, not just cope better with a life-altering disease?

I simply don’t buy Van Dam’s suggestion that to name names promotes quackery any more than I believe exposing anti-vaxxers promotes the anti-vaccine cause.

Is Van Dam only engaged in a polite discussion with fellow psychologists that needs to be strictly tone-policed to avoid offense or is he trying to reach, educate, and protect consumers as citizen scientists looking after their health and well-being? Maybe that is where we parted ways.

Will lessons in happiness solve the crisis in child mental health care?

bread and circussRome gave citizens bread and circuses. Is London giving citizens worthless randomized trials of inert interventions to solve the crisis of child mental health care without spending substantially more funds?

The UK  Department for Education (DfE) issued an Expression of Interest [ What’s that? ] for a large randomized trial comparing three preventive mental health interventions for promoting well-being among primary school children.

The three trialed interventions are:

Mindfulness

Mindfulness is the ability to direct attention to experience as it unfolds. It enables those who have learned it to be more able to be with their present experience, and respond more skilfully to whatever is happening. There is some evidence that it may be helpful in reducing anxiety, depressive symptoms and stress and improving wellbeing, attention, focus and cognitive skills.7 We know that mindfulness techniques are currently used by schools, with a range of existing programmes and approaches, but there is limited understanding of whether less intensive approaches work effectively in a school setting. The successful bidder will develop and trial a light touch (10-15 minute) intervention, comprising of simple exercises repeated at regular intervals (e.g. weekly or more than once a week) which provides teachers with materials to guide mindfulness practice e.g. audio tracks or guided exercises.

safety-net-PB-feeling-good-feeling-safe-resource-pack-a42Protective behaviours

Protective behaviours is a practical approach to personal safety, teaching children and young people to recognise early warning signs of not feeling safe and how to recognise where they can get help. It seeks to provide life skills, develop support structures and instil positive help seeking behaviours which can help keep children safe from a range of risks that may impact wellbeing and increase the risk of mental health problems. It is a well-established approach, with indications of ongoing use in schools, however evidence of effectiveness is limited. Some evidence suggests that it is beneficial for those at risk of mental health difficulties as well as the wider population, and it is relatively easy to integrate into the school environment. The successful bidder will develop and trial a light touch protective behaviours intervention which can easily be included in the school day, can be delivered by teachers/school staff to a whole class, with a small amount of training, and which builds on existing programmes and materials.

hands Relaxation and breathing-based techniques

Relaxation and breathing-based techniques and training for schools originated as targeted interventions to assist pupils with anxiety. However, there is emerging use of these approaches universally in primary schools, particularly in the form of short breathing exercises, with some reported increases in concentration, resilience, self-perception positivity and connection with others. There is currently limited evidence of wider use in schools or effectiveness, but there is a theoretical unpinning linking relaxation with improved wellbeing and engagement with learning. The successful bidder will develop and trial a light touch intervention that offers short regular exercises, delivered by teachers in the classroom with minimal training and materials, and which build on existing relation and breathing-based techniques.

Note that the requirement is that all three interventions be delivered in low intensity “light touch” versions, i.e, “easily be included in the school day, can be delivered by teachers/school staff to a whole class, with a small amount of training, and which builds on existing programmes and materials.”

The planned trial is ambitious and large-scale, involving:

  • Recruitment of 100 volunteer primary schools…representing a range of different school types, locations and demographics.
  • Even randomization of schools into one of three arms corresponding to the three interventions, with 33 schools in each arm.
  • Classes in each school evenly randomized to intervention or control group.
  • A small amount of funding would help cover costs of participation and to incentivise full engagement with the trial.:

Final selection of primary and secondary outcomes are left to applicants, but expected to include short measures of

  • Subjective Wellbeing
  • Mental health/psychological wellbeing
  • Engagement with education

The larger context

The expression of interest was a follow-up to “The Shared Society”, UK Prime Minister Theresa May’s recent speech at the Charity Commission. In that speech the Prime Minister identified “the burning injustice of mental illness” and stated:

“This is an historic opportunity to right a wrong, and give people deserving of compassion and support the attention and treatment they deserve. And for all of us to change the way we view mental illness so that striving to improve mental wellbeing is seen as just as natural, positive and good as striving to improve our physical wellbeing.”

However, the Independent noted:

The speech however barely announces any extra cash to improve underfunded services – with just an extra £15m expected to be pledged for creating “places of safety”. This amounts to about £23,000 per parliamentary constituency.

Research conducted by the Education Policy Institute Independent Commission on Children and Young People’s Mental Health in November found that a quarter of young people seeking mental health care are turned away by specialist services because of a lack of resources. Waiting times for treatment in many areas are also incredibly long.

The House of Commons Public Accounts Committee said in September that it was “sceptical” about the Government’s attempt to improve mental health services without a significant amount of extra cash.

Praise for the speech

Nonetheless, the Independent reported praise to the Prime Minister’s speech:

Paul Farmer, chief executive of Mind, the mental health charity, said it was good that the Prime Minister was talking about mental health.

“It’s important to see the Prime Minister talking about mental health and shows how far we have come in bringing the experiences of people with mental health problems up the political agenda,” he said.

“Mental health should be at the heart of government, and at the heart of society and communities – it’s been on the periphery for far too long.”

He said he welcomed the focus on prevention in schools and workplaces and support for people in crisis.

Sir Ian Cheshire, chairman of the Heads Together Campaign described the Prime Minister’s announcements as “extremely important and very welcome”.

“They show both a willingness to tackle the broad challenge of mental health support and a practical grasp of how to start making a real difference,” he said.

As I noted in another blog post, the Heads Together Campaign is an initiative of the Royals.

Paul Farmer, chief executive of Mind, the mental health charity, said it was good that the Prime Minister was talking about mental health.

“It’s important to see the Prime Minister talking about mental health and shows how far we have come in bringing the experiences of people with mental health problems up the political agenda,” he said.

“Mental health should be at the heart of government, and at the heart of society and communities – it’s been on the periphery for far too long.”

He said he welcomed the focus on prevention in schools and workplaces and support for people in crisis.

Sir Ian Cheshire, chairman of the Heads Together Campaign described the Prime Minister’s announcements as “extremely important and very welcome”.

“They show both a willingness to tackle the broad challenge of mental health support and a practical grasp of how to start making a real difference,” he said.

Praise for the interventions that were selected for evaluation

 An article in The Guardian reported praise for the interventions that were selected for evaluation:

Laura Henry, an early years consultant and Ofsted inspector, said the trials could save the government billions in social care and housing costs down the line. “I think it’s an excellent idea,” she said. “Over the last decade there has been a massive push to academia, results and school league tables and children’s personal social development has been left behind.

“A holistic approach is needed and children should be able to self-regulate their own behaviour.”

Henry, a former teacher whose elder son is on the autistic spectrum, said specially trained teachers should help with grieving techniques and that any questions about bullying and pupils’ friends needed to be sensitive.

“It’s absolutely the best way to spend DfE money,” she said. “It will save x amount of money in social care when they are adults.”

And:

The mindfulness trial was welcomed by the educational pioneer Sir Anthony Seldon, who was pooh-poohed when he brought in such classes while master of the private school Wellington College. He said: “It was negligent of government [in the past] to have this unintelligent response to wellbeing, saying this was la-la land and psychobabble. We have a crisis in mental health which is reducible now that government is beginning to take seriously the right interventions to look after the wellbeing of young people.”

Professor Alan Smithers, of the University of Buckingham, where Seldon is vice-chancellor, was more sceptical. He said: “It is good the government is having a trial and not rushing in. There are so many demands on resources for schools that it is important we know that mindfulness lessons work. “There are many calls on school funding: the need for teachers, the squeeze on budgets and school buildings.”

Lord Layard, Britain’s “happiness guru”, and Lord O’Donnell, the former cabinet secretary, will meet the government this month to discuss how to enable schools to measure children’s wellbeing as a guide to performance. They want schools to give similar weight to children’s happiness as to their academic results. Under their proposals, schools would be measured on whether pupils’ happiness improved or declined. Children would fill in a questionnaire asking, among other things, whether “I have at least one good friend”; “other people generally like me”; “other people pick on me or bully me”; and “I would rather be alone than with others”.

Pupils’ scores would be confidential, but could be used to alert the school to serious difficulties. NHS workers would provide psychological treatment to children in schools at short notice before they became so ill that they qualified for admission to mental health services.

The interventions are unlikely to improve mental health comes, even self-reported well-being, and may prove harmful.

I dare you to be positive Tssk! The UK has some talented mental health services researchers. Why aren’t we hearing their collective voices of outrage about a useless trial squandering millions of pounds, potentially harming schools and students, and mainly serving to distract from the government’s lack of action to correct the underfunding of both mental health care for children and the school systems?

Instead, we have some self-proclaimed authorities waxing enthusiastically. As a group, they are lacking in mental health training and serve to benefit immensely from these initiatives. Journalists should get them out of the picture or at least better reveal the conflicts of interest and balance their commentary with comments that are more evidence driven.

Even when delivered with full intensity, the interventions lack evidence of effectiveness needed to justify a large-scale trial. Yet The Expression of Interest specifies that they be delivered in a lite form – only a few minutes a week . This is Unlikely to improve the measured outcomes or impact on effective use of already scarce child mental health services with an acceptably long wait times.

Funding the trial is a poor substitute for better funding for mental health services and schools. Yet politicians and policymakers can point to them and argue that the UK is conducting the research desperately needed to address these issues and so we need to be patient.

I’m not sure we should consider these trials as serious attempts to contribute to the mental health services literature. Selection of the particular interventions to be trialed seems to be political and tied to what is already being done in some schools. Their existing implementation likely reflects vested interests that undoubtedly influenced the selection for trialing and hope to benefit financially from the opportunities it will provide. I don’t think that the mere fact that interventions are already in use justifies an ambitious and expensive evaluation of them unless there is further evidence that they are likely to be effective.

The Expression of Interest cites one review of mindfulness studies. I looked it up and it is unusually candid in indicating the limitations in quality and quantity of relevant studies that mindfulness training can affect such outcomes. The review stands in sharp contrast to the unbalanced and prematurely enthusiastic Mindful Nation UK report.

We should have serious concerns about the lack of evidence that Protective Behaviours could have any effect on the outcomes selected to evaluate the programs. Conceivably, it could do some harm to at risk children. Getting children to disclose bullying and frank abuse at home and school can only aggravate these problem and invite retaliation if effective intervention is not available to address these problems. I would be curious to know the extent to which primary school teachers are already aware of such problems but lack the tools or time to address them.

Basically, Protective Behaviours is a kind of screening program facilitated by encouragement to disclosed. Such programs can prove ineffective if they do not occur in a system prepared to quickly offer affective interventions. Such screening programs can compete for scarce resources that would otherwise be used to deal with already known problems requiring more intensive and focused intervention.

There is the precedent of GPs screening women for domestic abuse. Routine screening seemed to address a documented ignoring of the problem. However, the World Health Organization (WHO) withdrew the recommendation because of lack of any evidence that improved health outcomes for women and summoned consistent evidence that at least some women were harmed by ineffectual interventions that heighten the abuse that they were receiving.

The breathing and relaxation exercises might conceivably be a nonspecific control condition, except that all of the inventions are untried, lacking in evidence, and delivered in such a low intensity that they themselves are best nonspecific control conditions. I think it’s inconceivable that meaningful differences will be demonstrated among the three interventions. At best, the trials can conclude that they are equally effective or not effective at all. The question whether these interventions are better than other active interventions or other deployments of scarce resources left unaddressed.

eBook_PositivePsychology_345x550I will soon be offering e-books providing skeptical looks at mindfulness and positive psychology, as well as scientific writing courses on the web as I have been doing face-to-face for almost a decade.

Sign up at my new website to get advance notice of the forthcoming e-books and web courses, as well as upcoming blog posts at this and other blog sites.  Get advance notice of forthcoming e-books and web courses. Lots to see at CoyneoftheRealm.com.

Misleading systematic review of mindfulness studies used to promote Bensen Institute for Mind-Body Medicine services

A seriously flawed overview “systematic review “ of systematic reviews and meta-analyses of the effects of mindfulness on health and well-being alerts readers how they need to be skeptical of what they are told about the benefits of mindfulness.

Especially when the information comes those benefiting enormously from promoting the practice.

The glowing evaluation of the benefits of mindfulness presented in a PLOS One review is contradicted by a more comprehensive and systematic review which was cited but summarily dismissed. As we will see, the PLOS One article sidesteps substantial confirmation bias and untrustworthiness in the mindfulness literature.

The review was prepared by authors associated with the Benson-Henry Institute for Mind-Body Medicine, which is tied to Massachusetts General Hospital and Harvard Medical School. The institute directly markets mindfulness treatment to patients and training to professionals and organizations.  Its website provides links to research articles such as this one, which are used to market a wide range of programs –

being calm

Recently PLOS One published corrections to five articles from this group concerning previous statements about the authors having no conflicts of interest to declare. The corrections acknowledged extensive conflicts of interest.

The Competing Interests statement is incorrect. The correct Competing Interests statement is: The following authors hold or have held positions at the Benson-Henry Institute for Mind-Body Medicine at Massachusetts General Hospital, which is paid by patients and their insurers for running the SMART-3RP and related relaxation/mindfulness clinical programs, markets related products such as books, DVDs, CDs and the like, and holds a patent pending (PCT/US2012/049539 filed August 3, 2012) entitled “Quantitative Genomics of the Relaxation Response.”

While the review we will be discussing was not corrected, it should have been.

The same conflicts of interest should have been disclosed to readers evaluating the trustworthiness of what is being presented to them.

Probing this review will demonstrate just how hard it is to uncover the bias and distortions that routinely is provided by promoters of mindfulness wanting to demonstrate the evidence base for what they offer.

The article is

Gotink, R.A., Chu, P., Busschbach, J.J., Benson, H., Fricchione, G.L. and Hunink, M.M., 2015. Standardised mindfulness-based interventions in healthcare: an overview of systematic reviews and meta-analyses of RCTs. PLOS One, 10(4), p.e0124344.

The abstract offers the conclusion:

The evidence supports the use of MBSR and MBCT to alleviate symptoms, both mental and physical, in the adjunct treatment of cancer, cardiovascular disease, chronic pain, depression, anxiety disorders and in prevention in healthy adults and children.

This evaluation is more emphatically stated near the end of the article:

This review provides an overview of more trials than ever before and the intervention effect has thus been evaluated across a broad spectrum of target conditions, most of which are common chronic conditions. Study settings in many countries across the globe contributed to the analysis, further serving to increase the generalizability of the evidence. Beneficial effects were mostly seen in mental health outcomes: depression, anxiety, stress and quality of life improved significantly after training in MBSR or MBCT. These effects were seen both in patients with medical conditions and those with psychological disorders, compared with many types of control interventions (WL, TAU or AT). Further evidence for effectiveness was provided by the observed dose-response relationship: an increase in total minutes of practice and class attendance led to a larger reduction of stress and mood complaints in four reviews [18,20,37,54].

Are you impressed? “More than ever before”? “Generalizability of the evidence”? Really?

And in wrap up summary comments:

Although there is continued scepticism in the medical world towards MBSR and MBCT, the evidence indicates that MBSR and MBCT are associated with improvements in depressive symptoms, anxiety, stress, quality of life, and selected physical outcomes in the adjunct treatment of cancer, cardiovascular disease, chronic pain, chronic somatic diseases, depression, anxiety disorders, other mental disorders and in prevention in healthy adults and children.

Compare and contrast these conclusions with a more balanced and comprehensive review.

The US Agency for Healthcare Research and Quality (AHCRQ) commissioned a report from Johns Hopkins University Evidence-based Practice Center.

The 439 page report is publicly available:

Goyal M, Singh S, Sibinga EMS, Gould NF, Rowland-Seymour A, Sharma R, Berger Z, Sleicher D, Maron DD, Shihab HM, Ranasinghe PD, Linn S, Saha S, Bass EB, Haythornthwaite JA. Meditation Programs for Psychological Stress and Well-Being. Comparative Effectiveness Review No. 124. (Prepared by Johns Hopkins University Evidence-based Practice Center under Contract No. 290-2007-10061–I.) AHRQ Publication No. 13(14)-EHC116-EF. Rockville, MD: Agency for Healthcare Research and Quality; January 2014.

A companion, less detailed article was also published in JAMA: Internal Medicine:

Goyal, M., Singh, S., Sibinga, E.M., Gould, N.F., Rowland-Seymour, A., Sharma, R., Berger, Z., Sleicher, D., Maron, D.D., Shihab, H.M. and Ranasinghe, P.D., 2014. Meditation programs for psychological stress and well-being: a systematic review and meta-analysis. JAMA Internal Medicine, 174(3), pp.357-368.

Consider how conclusions of this article were characterized in the Bensen-Henry PLOS One article. The article is briefly mentioned without detailing its methods and conclusions.

Recently, Goyal et al. published a review of mindfulness interventions compared to active control and found significant improvements in depression and anxiety[7].

And

A recent review compared meditation to only active control groups, and although lower, also found a beneficial effect on depression, anxiety, stress and quality of life. This review was excluded in our study for its heterogeneity of interventions [7].

What the Goyal et JAMA: Internal Medicine actually said:

After reviewing 18 753 citations, we included 47 trials with 3515 participants. Mindfulness meditation programs had moderate evidence of improved anxiety (effect size, 0.38 [95% CI, 0.12-0.64] at 8 weeks and 0.22 [0.02-0.43] at 3-6 months), depression (0.30 [0.00-0.59] at 8 weeks and 0.23 [0.05-0.42] at 3-6 months), and pain (0.33 [0.03- 0.62]) and low evidence of improved stress/distress and mental health–related quality of life. We found low evidence of no effect or insufficient evidence of any effect of meditation programs on positive mood, attention, substance use, eating habits, sleep, and weight. We found no evidence that meditation programs were better than any active treatment (ie, drugs, exercise, and other behavioral therapies).

The review also notes that evidence of the effectiveness mindfulness interventions is largely limited to trials in which it is compared to no treatment, wait list, or a usually ill-defined treatment as usual (TAU).

In our comparative effectiveness analyses (Figure 1B), we found low evidence of no effect or insufficient evidence that any of the meditation programs were more effective than exercise, progressive muscle relaxation, cognitive-behavioral group therapy, or other specific comparators in changing any outcomes of interest. Few trials reported on potential harms of meditation programs. Of the 9 trials reporting this information, none reported any harms of the intervention.

This solid JAMA: Internal Medicine review explains why its conclusions may differ from past reviews:

Reviews to date report a small to moderate effect of mindfulness and mantra meditation techniques in reducing emotional symptoms (eg, anxiety, depression, and stress) and improving physical symptoms (eg, pain).7– 26 These reviews have largely included uncontrolled and controlled studies, and many of the controlled studies did not adequately control for placebo effects (eg, waiting list– or usual care–controlled studies). Observational studies have a high risk of bias owing to problems such as self-selection of interventions (people who believe in the benefits of meditation or who have prior experience with meditation are more likely to enroll in a meditation program and report that they benefited from one) and use of outcome measures that can be easily biased by participants’ beliefs in the benefits of meditation. Clinicians need to know whether meditation training has beneficial effects beyond self-selection biases and the nonspecific effects of time, attention, and expectations for improvement.27,28

Basically, this article insists that mindfulness be evaluated in a  head-to- head comparison to an active treatment. Failure to provide such a comparison means not being able to rule out that apparent effects of mindfulness are nonspecific, i.e.,  not due to any active ingredient of the practice.

An accompanying editorial commentary raised troubling issues about the state of the mindfulness literature. It noted that limiting inclusion to RCTs with an active control condition and a patient population experiencing mental or physical health problems left only 3% (47/18,753 of the citations that had been retrieved. Furthermore:

The modest benefit found in the study by Goyal et al begs the question of why, in the absence of strong scientifically vetted evidence, meditation in particular and complementary measures in general have become so popular, especially among the influential and well educated…What role is being played by commercial interests? Are they taking advantage of the public’s anxieties to promote use of complementary measures that lack a base of scientific evidence? Do we need to require scientific evidence of efficacy and safety for these measures?

How did the Bensen-Henry review arrive at a more favorable assessment?

The issue that dominated the solid Goyal et al systematic review and meta analysis is not prominent in the Bensen-Henry review. The latter article hardly mentions the importance of whether mindfulness is compared to an active treatment. It doesn’t mention if any difference in effect size for mindfulness can be expected when the comparison is an active treatment.

The Bensen-Henry review stated that it excluded systematic reviews and meta analyses if they did not focus on MBCT or MBSR. One has to search the supplementary materials to find that Goyal et al was excluded because it did not calculate separate effect sizes for mindfulness-based stress reduction (MBSR).

However, Bensen-Henry review included narrative systematic reviews that did not calculate effect sizes at all. Furthermore, the excluded Goyal et al JAMA: Internal Medicine article summarized MBSR separate from other forms of meditation and the more comprehensive AHCQR report provided detailed forest plots of effect sizes for MBSR with specific outcomes and patient populations.

Hmm, keeping out evidence that does fit with the sell-job story?

We need to keep in mind the poor manner in which MBSR was specified, particularly in the early studies that dominate the reviews covered by the Bensen – Henry article. Many of the treatments were not standardized and certainly not manualized. They sometimes, but not always incorporate psychoeducation, other cognitive behavioral techniques, and varying types of yoga.

The Bensen-Henry authors claimed to have performed quality assessments  of the reviews  included using a checklist based on the validated PRISMA guidelines. However, PRISMA evaluates the quality of reporting in reviews, not the quality of how the review was done. The checklist used by the Bensen-Henry authors was highly selective in terms of which PRISMA items it chose to include, left unvalidated, and simply eccentric. For instance, one item evaluated a review favorably if it interpreted studies “independent of funding source.”

A lack of independence of a study from its funding source is generally considered a high risk of bias.  There is ample documentation of  industry-funded studies and reviews exaggerating the efficacy of interventions supported by industry.

Our group received the Bill Silverman Prize from the Cochrane Collaboration for our identifying funding source as an overlooked source of bias in many meta analyses and, in particular, in Cochrane reviews. The Bensen-Henry checklist scores a review ignoring funding source as a virtue, not a vice! These authors are letting trials and reviews from promoters of mindfulness off the hook for potential conflict of interest, including their own studies and this review.

Examination of the final sample of reviews included in the Bensen-Henry analysis reveals that some are narrative reviews and could not contribute effect sizes. Some are older reviews that depend on a less developed literature. While optimistic about the promise of mindfulness, authors of these reviews frequently complained about the limits on the quantity and quality of available studies, calling for larger and better quality studies. When integrated and summarized by the Bensen-Henry authors, these reviews were given a more positive glow than the original authors conveyed.

Despite claims of being an “overview of more trials than ever before”, Bensen-Henry excluded all but 23 reviews. Some of those included do not appear to be recent or rigorous, particularly when contrasted with the quality and rigor of the excluded Goyal et al:

MJ, Norris RL, Bauer-Wu SM (2006) Mindfulness meditation for oncology patients: A discussion and critical review. Integr Cancer Ther 5: 98–108. pmid:16685074

Shennan C, Payne S, Fenlon D (2011) What is the evidence for the use of mindfulness-based interventions in cancer care? A review. Psycho-Oncology 20: 681–697.

Veehof MM, Oskam MJ, Schreurs KMG, Bohlmeijer ET (2011) Acceptance-based interventions for the treatment of chronic pain: A systematic review and meta-analysis. Pain 152: 533–542

Coelho HF, Canter PH, Ernst E (2007) Mindfulness-Based Cognitive Therapy: Evaluating Current Evidence and Informing Future Research. J Consult Clin Psychol 75: 1000–1005.

Ledesma D, Kumano H (2009) Mindfulness-based stress reduction and cancer: A meta-analysis. Psycho-Oncology 18: 571–579.

Ott MJ, Norris RL, Bauer-Wu SM (2006) Mindfulness meditation for oncology patients: A discussion and critical review. Integr Cancer Ther 5: 98–108.

Burke CA (2009) Mindfulness-Based Approaches with Children and Adolescents: A Preliminary Review of Current Research in an Emergent Field. J Child Fam Stud.

Do we get the most authoritative reviews of mindfulness from  Holist Nurs Pract, Integr Cancer Ther, and Psycho-Oncology?

To cite just one example of the weakness of evidence being presented as strong, take the bold Bensen-Henry conclusion:

Further evidence for effectiveness was provided by the observed dose-response relationship: an increase in total minutes of practice and class attendance led to a larger reduction of stress and mood complaints in four reviews [18,20,37,54].

“Observed dose-response relationship”? This claim is  based [check out with respect to the citations just above] on Ott et al, 18, Smith et al 20, Burke 37 and Proulx 54, which makes the evidence neither recent nor systematic. I am confident that other examples will not hold up if scrutinized.

Further contradiction of the too perfect picture of mindfulness therapy conveyed by the Bensen – Henry review.

A more recent PLOS One review of mindfulness studies exposed the confirmation bias in the published mindfulness literature. It suggested a too perfect picture has been created of uniformly positive studies.

Coronado-Montoya, S., Levis, A.W., Kwakkenbos, L., Steele, R.J., Turner, E.H. and Thombs, B.D., 2016. Reporting of positive results in randomized controlled trials of mindfulness-based mental health interventions. PLOS One, 11(4), p.e0153220.

A systematic search yielded 124 RCTs of mindfulness-based treatments:

108 (87%) of 124 published trials reported >1 positive outcome in the abstract, and 109(88%) concluded that mindfulness-based therapy was effective, 1.6 times greater than the expected number of positive trials based on effect size d = 0.55 (expected number positivetrials = 65.7). Of 21 trial registrations, 13 (62%) remained unpublished 30 months post-trial completion.

Furthermore:

None of the 21 registrations, however, adequately specified a single primary outcome (or multiple primary outcomes with an appropriate plan for statistical adjustment) and specified the outcome measure, the time of assessment, and the metric (e.g., continuous, dichotomous). When we removed the metric requirement, only 2 (10%) registrations were classified as adequate.

And finally:

There were only 3 trials that were presented unequivocally as negative trials without alternative interpretations or caveats to mitigate the negative results and suggest that the treatment might still be an effective treatment.

What we have is a picture of trials of mindfulness-based treatment having an excess of positive studies, given the study sample sizes. Selective reporting of positive outcomes likely contributed to this excess of published positive findings in the published literature. Most of the trials were not preregistered and so it’s unclear whether the positive outcomes that were reported were hypothesized to be the primary outcomes of interest. Most of the trials that were preregistered remained unpublished 30 months after the trials were completed.

The Goyal et al. study originally planned to conduct quantitative analyses of publication biases, but abandoned the effort when they couldn’t find sufficient numbers of the 47 studies that that reported most of the outcomes they evaluated.

Conclusion

 The Bensen-Henry review produces a glowing picture of the quality of RCTs evaluating MSBR and the consistency of positive findings across diverse outcomes and populations. This is consistent with the message that they want to promote in marketing their products to patients, clinicians, and institutions. In this blog post I’ve uncovered substantial problems in internal to the Bensen-Henry review in terms of the studies that were included and the manner in which they were evaluated. But now we have external evidence in two reviews without obvious conflicts of interest come into markedly different appraisals of a literature that lacks appropriate control groups and seems to be reporting findings with a distinct confirmation bias.

I could have gone further, but what I found about the Bensen-Henry review seems sufficient for a serious challenge to the validity of its conclusions.  Investigation of the claims made about dose-response relationships between amount of mindfulness practice and outcomes should encourage probing of other specific claims.

The larger issue is that we should not rely on promoters of MSBR products to provide unbiased estimates of their efficacy. This issue recalls very similar problems in the evaluation of Triple P Parenting Programs. Evaluations in which promoters were involved produce markedly more positive results than from independent evaluations. Exposure by my colleagues and me led to over 50 corrections and corrigendum to articles that previously had no conflicts of interest. But the process did not occur without fierce resistance from those whose livelihood was being challenged.

A correction to the Bensen-Henry PLOS One review is in order to clarify the obvious conflicts of interest of the authors. But the problem is not limited to reviews or original studies from Benson-Henry Institute for Mind-Body Medicine. It’s time that authors be required to answer more explicit questions about conflict of interest. Ruling out a conflict of interest should be based on authors having to endorse explicitly no conflicts, rather than on their basis of their not disclosing a conflict and then being able to claim it was an oversight that they did not report one.

Postscript Who was watching at PLOS One to keep out infomercials from promoters associated with Massachusetts General Hospital and Harvard Medical School? The Academic Editor was To avoid the appearance of  a conflict of interest,  should he have recused him from serving as editor?

This is another flawed paper for which I’d love to see the reviews.

eBook_Mindfulness_345x550I will soon be offering e-books providing skeptical looks at mindfulness and positive psychology, as well as scientific writing courses on the web as I have been doing face-to-face for almost a decade.

Sign up at my new website to get advance notice of the forthcoming e-books and web courses, as well as upcoming blog posts at this and other blog sites.  Get advance notice of forthcoming e-books and web courses. Lots to see at CoyneoftheRealm.com.

 

Unintended consequences of universal mindfulness training for schoolchildren?

the mindful nationThis is the first installment of what will be a series of occasional posts about the UK Mindfulness All Party Parliamentary Group report,  Mindful Nation.

  • Mindful Nation is seriously deficient as a document supposedly arguing for policy based on evidence.
  • The professional and financial interests of lots of people involved in preparation of the document will benefit from implementation of its recommendations.
  • After an introduction, I focus on two studies singled in Mindful Nation out as offering support for the benefits of mindfulness training for school children.
  • Results of the group’s cherrypicked studies do not support implementation of mindfulness training in the schools, but inadvertently highlight some issues.
  • Investment in universal mindfulness training in the schools is unlikely to yield measurable, socially significant results, but will serve to divert resources from schoolchildren more urgently in need of effective intervention and support.
  • Mindfulness Nation is another example of  delivery of  low intensity  services to mostly low risk persons to the detriment of those in greatest and most urgent need.

The launch event for the Mindful Nation report billed it as the “World’s first official report” on mindfulness.

Mindful Nation is a report written by the UK Mindfulness All-Party Parliamentary Group.

The Mindfulness All-Party Parliamentary Group (MAPPG)  was set up to:

  • review the scientific evidence and current best practice in mindfulness training
  • develop policy recommendations for government, based on these findings
  • provide a forum for discussion in Parliament for the role of mindfulness and its implementation in public policy.

The Mindfulness All-Party Parliamentary Group describes itself as

Impressed by the levels of both popular and scientific interest, and launched an inquiry to consider the potential relevance of mindfulness to a range of urgent policy challenges facing government.

Don’t get confused by this being a government-commissioned report. The report stands in sharp contrast to one commissioned by the US government in terms of unbalanced constitution of the committee undertaking the review, and lack  of transparency in search for relevant literature,  and methodology for rating and interpreting of the quality of available evidence.

ahrq reportCompare the claims of Mindful Nation to a comprehensive systematic review and meta-analysis prepared for the US Agency for Healthcare Research and Quality (AHRQ) that reviewed 18,753 citations, and found only 47 trials (3%) that included an active control treatment. The vast majority of studies available for inclusion had only a wait list or no-treatment control group and so exaggerated any estimate of the efficacy of mindfulness.

Although the US report was available to those  preparing the UK Mindful Nation report, no mention is made of either the full contents of report or a resulting publication in a peer-reviewed journal. Instead, the UK Mindful Nation report emphasized narrative and otherwise unsystematic reviews, and meta-analyses not adequately controlling for bias.

When the abridged version of the AHRQ report was published in JAMA: Internal Medicine, an accompanying commentary raises issues even more applicable to the Mindful Nation report:

The modest benefit found in the study by Goyal et al begs the question of why, in the absence of strong scientifically vetted evidence, meditation in particular and complementary measures in general have become so popular, especially among the influential and well educated…What role is being played by commercial interests? Are they taking advantage of the public’s anxieties to promote use of complementary measures that lack a base of scientific evidence? Do we need to require scientific evidence of efficacy and safety for these measures?

The members of the UK Mindfulness All-Party Parliamentary Group were selected for their positive attitude towards mindfulness. The collection of witnesses they called to hearings were saturated with advocates of mindfulness and those having professional and financial interests in arriving at a positive view. There is no transparency in terms of how studies or testimonials were selected, but the bias is notable. Many of the scientific studies were methodologically poor, if there was any methodology at all. Many were strongly stated, but weakly substantiated opinion pieces. Authors often included those having  financial interests in obtaining positive results, but with no acknowledgment of conflict of interest. The glowing testimonials were accompanied by smiling photos and were unanimous in their praise of the transformative benefits of mindfulness.

As Mark B. Cope and David B. Allison concluded about obesity research, such a packing of the committee and a highly selective review of the literature leads to a ”distortion of information in the service of what might be perceived to be righteous ends.” [I thank Tim Caulfield for calling this quote to my attention].

Mindfulness in the schools

The recommendations of Mindfulness Nation are

  1. The Department for Education (DfE) should designate, as a first step, three teaching schools116 to pioneer mindfulness teaching,co-ordinate and develop innovation, test models of replicability and scalability and disseminate best practice.
  2. Given the DfE’s interest in character and resilience (as demonstrated through the Character Education Grant programme and its Character Awards), we propose a comparable Challenge Fund of £1 million a year to which schools can bid for the costs of training teachers in mindfulness.
  3. The DfE and the Department of Health (DOH) should recommend that each school identifies a lead in schools and in local services to co-ordinate responses to wellbeing and mental health issues for children and young people117. Any joint training for these professional leads should include a basic training in mindfulness interventions.
  4. The DfE should work with voluntary organisations and private providers to fund a freely accessible, online programme aimed at supporting young people and those who work with them in developing basic mindfulness skills118.
Payoff of Mindful Nation to Oxford Mindfulness Centre will be huge.
Payoff of Mindful Nation to Oxford Mindfulness Centre will be huge.

Leading up to these recommendations, the report outlined an “alarming crisis” in the mental health of children and adolescents and proposes:

Given the scale of this mental health crisis, there is real urgency to innovate new approaches where there is good preliminary evidence. Mindfulness fits this criterion and we believe there is enough evidence of its potential benefits to warrant a significant scaling-up of its availability in schools.

Think of all the financial and professional opportunities that proponents of mindfulness involved in preparation of this report have garnered for themselves.

Mindfulness to promote executive functioning in children and adolescents

For the remainder of the blog post, I will focus on the two studies cited in support of the following statement:

What is of particular interest is that those with the lowest levels of executive control73 and emotional stability74 are likely to benefit most from mindfulness training.

The terms “executive control” and “emotional stability” were clarified:

Many argue that the most important prerequisites for child development are executive control (the management of cognitive processes such as memory, problem solving, reasoning and planning) and emotion regulation (the ability to understand and manage the emotions, including and especially impulse control). These main contributors to self-regulation underpin emotional wellbeing, effective learning and academic attainment. They also predict income, health and criminality in adulthood69. American psychologist, Daniel Goleman, is a prominent exponent of the research70 showing that these capabilities are the biggest single determinant of life outcomes. They contribute to the ability to cope with stress, to concentrate, and to use metacognition (thinking about thinking: a crucial skill for learning). They also support the cognitive flexibility required for effective decision-making and creativity.

Actually, Daniel Goleman is the former editor of the pop magazine Psychology Today and an author of numerous pop books.

The first cited paper.

73 Flook L, Smalley SL, Kitil MJ, Galla BM, Kaiser-Greenland S, Locke J, et al. Effects of mindful  awareness practices on executive functions in elementary school children. Journal of Applied School Psychology. 2010;26(1):70-95.

Journal of Applied School Psychology is a Taylor-Francis journal, formerly known as Special Services in the Schools (1984 – 2002).  Its Journal Impact Factor is 1.30.

One of the authors of the article, Susan Kaiser-Greenland is a mindfulness entrepreneur as seen in her website describing her as an author, public speaker, and educator on the subject of sharing secular mindfulness and meditation with children and families. Her books are The Mindful Child: How to Help Your Kid Manage Stress and Become Happier, Kinder, and More Compassionate and Mindful Games: Sharing Mindfulness and Meditation with Children, Teens, and Families and the forthcoming The Mindful Games Deck: 50 Activities for Kids and Teens.

This article represents the main research available on Kaiser-Greenfield’s Inner Kids program and figures prominently in her promotion of her products.

The sample consisted of 64 children assigned to either mindful awareness practices (MAPs; n = 32) or a control group consisting of a silent reading period (n = 32).

The MAPs training used in the current study is a curriculum developed by one of the authors (SKG). The program is modeled after classical mindfulness training for adults and uses secular and age appropriate exercises and games to promote (a) awareness of self through sensory awareness (auditory, kinesthetic, tactile, gustatory, visual), attentional regulation, and awareness of thoughts and feelings; (b) awareness of others (e.g., awareness of one’s own body placement in relation to other people and awareness of other people’s thoughts and feelings); and (c) awareness of the environment (e.g., awareness of relationships and connections between people, places, and things).

A majority of exercises involve interactions among students and between students and the instructor.

Outcomes.

The primary EF outcomes were the Metacognition Index (MI), Behavioral Regulation Index (BRI), and Global Executive Composite (GEC) as reported by teachers and parents

Wikipedia presents the results of this study as:

The program was delivered for 30 minutes, twice per week, for 8 weeks. Teachers and parents completed questionnaires assessing children’s executive function immediately before and following the 8-week period. Multivariate analysis of covariance on teacher and parent reports of executive function (EF) indicated an interaction effect baseline EF score and group status on posttest EF. That is, children in the group that received mindful awareness training who were less well regulated showed greater improvement in EF compared with controls. Specifically, those children starting out with poor EF who went through the mindful awareness training showed gains in behavioral regulation, metacognition, and overall global executive control. These results indicate a stronger effect of mindful awareness training on children with executive function difficulties.

The finding that both teachers and parents reported changes suggests that improvements in children’s behavioral regulation generalized across settings. Future work is warranted using neurocognitive tasks of executive functions, behavioral observation, and multiple classroom samples to replicate and extend these preliminary findings.”

What I discovered when I scrutinized the study.

 This study is unblinded, with students and their teachers and parents providing the subjective ratings of the students well aware of which group students are assigned. We are not given any correlations among or between their ratings and so we don’t know whether there is just a global subjective factor (easy or difficult child, well-behaved or not) operating for either teachers or parents, or both.

It is unclear for what features of the mindfulness training the comparison reading group offers control or equivalence. The two groups are  different in positive expectations and attention and support that are likely to be reflected the parent and teacher ratings. There’s a high likelihood of any differences in outcomes being nonspecific and not something active and distinct ingredient of mindfulness training. In any comparison with the students assigned to reading time, students assigned to mindfulness training have the benefit of any active ingredient it might have, as well as any nonspecific, placebo ingredients.

This is exceedingly weak design, but one that dominates evaluations of mindfulness.

With only 32 students per group, note too that this is a seriously underpowered study. It has less than a 50% probability of detecting a moderate sized effect if one is present. And because of the larger effect size needed to achieve statistical significance with such a small sample size, and statistically significant effects will be large, even if unlikely to replicate in a larger sample. That is the paradox of low sample size we need to understand in these situations.

Not surprisingly, there were no differences between the mindfulness and reading control groups on any outcomes variable, whether rated by parents or teachers. Nonetheless, the authors rescued their claims for an effective intervention with:

However, as shown by the significance of interaction terms, baseline levels of EF (GEC reported by teachers) moderated improvement in posttest EF for those children in the MAPs group compared to children in the control group. That is, on the teacher BRIEF, children with poorer initial EF (higher scores on BRIEF) who went through MAPs training showed improved EF subsequent to the training (indicated by lower GEC scores at posttest) compared to controls.

Similar claims were made about parent ratings. But let’s look at figure 3 depicting post-test scores. These are from the teachers, but results for the parent ratings are essentially the same.

teacher BRIEF quartiles

Note the odd scaling of the X axis. The data are divided into four quartiles and then the middle half is collapsed so that there are three data points. I’m curious about what is being hidden. Even with the sleight-of-hand, it appears that scores for the intervention and control groups are identical except for the top quartile. It appears that just a couple of students in the control group are accounting for any appearance of a difference. But keep in mind that the upper quartile is only a matter of eight students in each group.

This scatter plot is further revealing:

teacher BRIEF

It appears that the differences that are limited to the upper quartile are due to a couple of outlier control students. Without them, even the post-hoc differences that were found in the upper quartile between intervention control groups would likely disappear.

Basically what we are seeing is that most students do not show any benefit whatsoever from mindfulness training over being in a reading group. It’s not surprising that students who were not particularly elevated on the variables of interest do not register an effect. That’s a common ceiling effect in such universally delivered interventions in general population samples

Essentially, if we focus on the designated outcome variables, we are wasting the students’ time as well as that of the staff. Think of what could be done if the same resources could be applied in more effective ways. There are a couple of students in in this study were outliers with low executive function. We don’t know how else they otherwise differ.Neither in the study, nor in the validation of these measures is much attention given to their discriminant validity, i.e., what variables influence the ratings that shouldn’t. I suspect strongly that there are global, nonspecific aspects to both parent and teacher ratings such that they are influenced by the other aspects of these couple of students’ engagement with their classroom environment, and perhaps other environments.

I see little basis for the authors’ self-congratulatory conclusion:

The present findings suggest that mindfulness introduced in a general  education setting is particularly beneficial for children with EF difficulties.

And

Introduction of these types of awareness practices in elementary education may prove to be a viable and cost-effective way to improve EF processes in general, and perhaps specifically in children with EF difficulties, and thus enhance young children’s socio-emotional, cognitive, and academic development.

Maybe the authors stared with this conviction and it was unshaken by disappointing findings.

Or the statement made in Mindfulness Nation:

What is of particular interest is that those with the lowest levels of executive control73 and emotional stability74 are likely to benefit most from mindfulness training.

But we have another study that is cited for this statement.

74. Huppert FA, Johnson DM. A controlled trial of mindfulness training in schools: The importance of practice for an impact on wellbeing. The Journal of Positive Psychology. 2010; 5(4):264-274.

The first author, Felicia Huppert is a  Founder and Director – Well-being Institute and Emeritus Professor of Psychology at University of Cambridge, as well as a member of the academic staff of the Institute for Positive Psychology and Education of the Australian Catholic University.

This study involved 173 14- and 15- year old  boys from a private Catholic school.

The Journal of Positive Psychology is not known for its high methodological standards. A look at its editorial board suggests a high likelihood that manuscripts submitted will be reviewed by sympathetic reviewers publishing their own methodologically flawed studies, often with results in support of undeclared conflicts of interest.

The mindfulness training was based on the program developed by Kabat-Zinn and colleagues at the University of Massachusetts Medical School (Kabat-Zinn, 2003). It comprised four 40 minute classes, one per week, which presented the principles and practice of mindfulness meditation. The mindfulness classes covered the concepts of awareness and acceptance, and the mindfulness practices included bodily awareness of contact points, mindfulness of breathing and finding an anchor point, awareness of sounds, understanding the transient nature of thoughts, and walking meditation. The mindfulness practices were built up progressively, with a new element being introduced each week. In some classes, a video clip was shown to highlight the practical value of mindful awareness (e.g. “The Last Samurai”, “Losing It”). Students in the mindfulness condition were also provided with a specially designed CD, containing three 8-minute audio files of mindfulness exercises to be used outside the classroom. These audio files reflected the progressive aspects of training which the students were receiving in class. Students were encouraged to undertake daily practice by listening to the appropriate audio files. During the 4-week training period, students in the control classes attended their normal religious studies lessons.

A total of 155 participants had complete data at baseline and 134 at follow-up (78 in the mindfulness and 56 in the control condition). Any student who had missing data are at either time point was simply dropped from the analysis. The effects of this statistical decison are difficult to track in the paper. Regardless, there was a lack of any difference between intervention and control group and any of a host of outcome variables, with none designated as primary outcome.

Actual practicing of mindfulness by students was inconsistent.

One third of the group (33%) practised at least three times a week, 34.8% practised more than once but less than three times a week, and 32.7% practised once a week or less (of whom 7 respondents, 8.4%, reported no practice at all). Only two students reported practicing daily. The practice variable ranged from 0 to 28 (number of days of practice over four weeks). The practice variable was found to be highly skewed, with 79% of the sample obtaining a score of 14 or less (skewness = 0.68, standard error of skewness = 0.25).

The authors rescue their claim of a significant effect for the mindfulness intervention with highly complex multivariate analyses with multiple control variables in which outcomes within-group effects for students assigned to mindfulness  were related to the extent of students actually practicing mindfulness. Without controlling for the numerous (and post-hoc) multiple comparisons, results were still largely nonsignificant.

One simple conclusion that can be drawn is that despite a lot of encouragement, there was little actual practice of mindfulness by the relatively well-off students in a relatively highly resourced school setting. We could expect results to improve with wider dissemination to schools with less resources and less privileged students.

The authors conclude:

The main finding of this study was a significant improvement on measures of mindfulness and psychological well-being related to the degree of individual practice undertaken outside the classroom.

Recall that Mindful Nation cited the study in the following context:

What is of particular interest is that those with the lowest levels of executive control73 and emotional stability74 are likely to benefit most from mindfulness training.

These are two methodologically weak studies with largely null findings. They are hardly the basis for launching a national policy implementing universal mindfulness in the schools.

As noted in the US AHRQ report, despite a huge number of studies of mindfulness having been conducted, few involved a test with an adequate control group, and so there’s little evidence that mindfulness has any advantage over any active treatment. Neither of these studies disturbed that conclusion, although they are spun both in the original studies and in the Mindful Nation report to be positive. Both papers were published in journals where the reviewers were likely to be overly sympathetic and not at him tentative to serious methodological and statistical problems.

The committee writing Mindful Nation arrived at conclusions consistent with their prior enthusiasm for mindfulness and their vested interest in it. They sorted through evidence to find what supported their pre-existing assumptions.

Like UK resilience programs, the recommendations of Mindful Nation put considerable resources in the delivery of services to a large population and likely to have the threshold of need to register a socially in clinically significant effect. On a population level, results of the implementation are doomed to fall short of its claims. Those many fewer students in need more timely, intensive, and tailored services are left underserved. Their presence is ignored or, worse, invoked to justify the delivery of services to the larger group, with the needy students not benefiting.

In this blog post, I mainly focused on two methodologically poor studies. But for the selection of these particular studies, I depended on the search of the authors of Mindful Nation and the emphasis that were given to these two studies for some sweeping claims in the report. I will continue to be writing about the recommendations of Mindful Nation. I welcome reader feedback, particularly from readers whose enthusiasm for mindfulness is offended. But I urge them not simply to go to Google and cherry pick an isolated study and ask me to refute its claims.

Rather, we need to pay attention to the larger literature concerning mindfulness, its serious methodological problems, and the sociopolitical forces and vested interests that preserve a strong confirmation bias, both in the “scientific” literature and its echoing in documents like Mindful Nation.

Why PhD students should not evaluate a psychotherapy for their dissertation project

  • Things some clinical and health psychology students wish they had known before they committed themselves to evaluating a psychotherapy for their dissertation study.
  • A well designed pilot study addressing feasibility and acceptability issues in conducting and evaluating psychotherapies is preferable to an underpowered study which won’t provide a valid estimate of the efficacy of the intervention.
  • PhD students would often be better off as research parasites – making use of existing published data – rather than attempting to organize their own original psychotherapy study, if their goal is to contribute meaningfully to the literature and patient care.
  • Reading this blog, you will encounter a link to free, downloadable software that allows you to make quick determinations of the number of patients needed for an adequately powered psychotherapy trial.

I so relish the extra boost of enthusiasm that many clinical and health psychology students bring to their PhD projects. They not only want to complete a thesis of which they can be proud, they want their results to be directly applicable to improving the lives of their patients.

Many students are particularly excited about a new psychotherapy about which extravagant claims are being made that it’s better than its rivals.

I have seen lots of fad and fashions come and go, third wave, new wave, and no wave therapies. When I was a PhD student, progressive relaxation was in. Then it died, mainly because it was so boring for therapists who had to mechanically provide it. Client centered therapy was fading with doubts that anyone else could achieve the results of Carl Rogers or that his three facilitative conditions of unconditional positive regard, genuineness,  and congruence were actually distinguishable enough to study.  Gestalt therapy was supercool because of the charisma of Fritz Perls, who distracted us with his showmanship from the utter lack of evidence for its efficacy.

I hate to see PhD students demoralized when their grand plans prove unrealistic.  Inevitably, circumstances force them to compromise in ways that limit any usefulness to their project, and maybe even threaten their getting done within a reasonable time period. Overly ambitious plans are the formidable enemy of the completed dissertation.

The numbers are stacked against a PhD student conducting an adequately powered evaluation of a new psychotherapy.

This blog post argues against PhD students taking on the evaluation of a new therapy in comparison to an existing one, if they expect to complete their projects and make meaningful contribution to the literature and to patient care.

I’ll be drawing on some straightforward analysis done by Pim Cuijpers to identify what PhD students are up against when trying to demonstrate that any therapy is better than treatments that are already available.

Pim has literally done dozens of meta-analyses, mostly of treatments for depression and anxiety. He commands a particular credibility, given the quality of this work. The way Pim and his colleagues present a meta-analysis is so straightforward and transparent that you can readily examine the basis of what he says.

Disclosure: I collaborated with Pim and a group of other authors in conducting a meta-analysis as to whether psychotherapy was better than a pill placebo. We drew on all the trials allowing a head-to-head comparison, even though nobody ever really set out to pit the two conditions against each other as their first agenda.

Pim tells me that the brief and relatively obscure letter, New Psychotherapies for Mood and Anxiety Disorders: Necessary Innovation or Waste of Resources? on which I will draw is among his most unpopular pieces of work. Lots of people don’t like its inescapable message. But I think that if PhD students should pay attention, they might avoid a lot of pain and disappointment.

But first…

Note how many psychotherapies have been claimed to be effective for depression and anxiety. Anyone trying to make sense of this literature has to contend with claims being based on a lot of underpowered trials– too small in sample size to be expected reasonably to detect the effects that investigators claim – and that are otherwise compromised by methodological limitations.

Some investigators were simply naïve about clinical trial methodology and the difficulties doing research with clinical populations. They may have not understand statistical power.

But many psychotherapy studies end up in bad shape because the investigators were unrealistic about the feasibility of what they were undertaken and the low likelihood that they could recruit the patients in the numbers that they had planned in the time that they had allotted. After launching the trial, they had to change strategies for recruitment, maybe relax their selection criteria, or even change the treatment so it was less demanding of patients’ time. And they had to make difficult judgments about what features of the trial to drop when resources ran out.

Declaring a psychotherapy trial to be a “preliminary” or a “pilot study” after things go awry

The titles of more than a few articles reporting psychotherapy trials contain the apologetic qualifier after a colon: “a preliminary study” or “a pilot study”. But the studies weren’t intended at the outset to be preliminary or pilot studies. The investigators are making excuses post-hoc – after the fact – for not having been able to recruit sufficient numbers of patients and for having had to compromise their design from what they had originally planned. The best they can hope is that the paper will somehow be useful in promoting further research.

Too many studies from which effect sizes are entered into meta-analyses should have been left as pilot studies and not considered tests of the efficacy of treatments. The rampant problem in the psychotherapy literature is that almost no one treats small scale trials as mere pilot studies. In a recent blog post, I provided readers with some simple screening rules to identify meta-analyses of psychotherapy studies that they could dismiss from further consideration. One was whether there were sufficient numbers of adequately powered studies,  Often there are not.

Readers take their inflated claims of results of small studies seriously, when these estimates should be seen as unrealistic and unlikely to be replicated, given a study’s sample size. The large effect sizes that are claimed are likely the product of p-hacking and the confirmation bias required to get published. With enough alternative outcome variables to choose from and enough flexibility in analyzing and interpreting data, almost any intervention can be made to look good.

The problem is is readily seen in the extravagant claims about acceptance and commitment therapy (ACT), which are so heavily dependent on small, under-resourced studies supervised by promoters of ACT that should not have been used to generate effect sizes.

Back to Pim Cuijpers’ brief letter. He argues, based on his numerous meta-analyses, that it is unlikely that a new treatment will be substantially more effective than an existing credible, active treatment.  There are some exceptions like relaxation training versus cognitive behavior therapy for some anxiety disorders, but mostly only small differences of no more than d= .20 are found between two active, credible treatments. If you search the broader literature, you can find occasional exceptions like CBT versus psychoanalysis for bulimia, but most you find prove to be false positives, usually based on investigator bias in conducting and interpreting a small, underpowered study.

You can see this yourself using the freely downloadable G*power program and plug in d= 0.20 for calculating the number of patients needed for a study. To be safe, add more patients to allow for the expectable 25% dropout rate that has occurred across trials. The number you get would require a larger study than has ever been done in the past, including the well-financed NIMH Collaborative trial.

G power analyses

Even more patients would be needed for the ideal situation in which a third comparison group allowed  the investigator to show the active comparison treatment had actually performed better than a nonspecific treatment that was delivered with the same effectiveness that the other had shown in earlier trials. Otherwise, a defender of the established therapy might argue that the older treatment had not been properly implemented.

So, unless warned off, the PhD student plans a study to show not only that now hypothesis can be rejected that the new treatment is no better than the existing one, but that in the same study the existing treatment had been shown to be better than wait list. Oh my, just try to find an adequately powered, properly analyzed example of a comparison of two active treatments plus a control comparison group in the existing published literature. The few examples of three group designs in which a new psychotherapy had come out better than an effectively implemented existing treatment are grossly underpowered.

These calculations so far have all been based on what would be needed to reject the null hypothesis of no difference between the active treatment and a more established one. But if the claim is that the new treatment is superior to the existing treatment, our PhD student now needs to conduct a superiority trial in which some criteria is pre-set (such as greater than a moderate difference, d= .30) and the null hypothesis is that the advantage of the new treatment is less. We are now way out into the fantasyland of breakthrough, but uncompleted dissertation studies.

Two take away messages

 The first take away message is that we should be skeptical of claims of the new treatment is better than past ones except when the claim occurs in a well-designed study with some assurance that it is free of investigator bias. But the claim also has to arise in a trial that is larger than almost any psychotherapy study is ever been done. Yup, most comparative psychotherapy studies are underpowered and we cannot expect robust claims are robust that one treatment is superior to another.

But for PhD students been doing a dissertation project, the second take away message is that they should not attempt to show that one treatment is superior to another in the absence of resources they probably don’t have.

The psychotherapy literature does not need another study with too few patients to support its likely exaggerated claims.

An argument can be made that it is unfair and even unethical to enroll patients in a psychotherapy RCT with insufficient sample size. Some of the patients will be randomized to the control condition that is not what attracted them to the trial. All of the patients will be denied having been in a trial makes a meaningful contribution to the literature and to better care for patients like themselves.

What should the clinical or health psychology PhD student do, besides maybe curb their enthusiasm? One opportunity to make meaningful contributions to literature by is by conducting small studies testing hypotheses that can lead to improvement in the feasibility or acceptability of treatments to be tested in studies with more resources.

Think of what would’ve been accomplished if PhD students had determined in modest studies that it is tough to recruit and retain patients in an Internet therapy study without some communication to the patients that they are involved in a human relationship – without them having what Pim Cuijpers calls supportive accountability. Patients may stay involved with the Internet treatment when it proves frustrating only because they have the support and accountability to someone beyond their encounter with an impersonal computer. Somewhere out there, there is a human being who supports them and sticking it out with the Internet psychotherapy and will be disappointed if they don’t.

A lot of resources have been wasted in Internet therapy studies in which patients have not been convinced that what they’re doing is meaningful and if they have the support of a human being. They drop out or fail to do diligently any homework expected of them.

Similarly, mindfulness studies are routinely being conducted without anyone establishing that patients actually practice mindfulness in everyday life or what they would need to do so more consistently. The assumption is that patients assigned to the mindfulness diligently practice mindfulness daily. A PhD student could make a valuable contribution to the literature by examining the rates of patients actually practicing mindfulness when the been assigned to it in a psychotherapy study, along with barriers and facilitators of them doing so. A discovery that the patients are not consistently practicing mindfulness might explain weaker findings than anticipated. One could even suggest that any apparent effects of practicing mindfulness were actually nonspecific, getting all caught up in the enthusiasm of being offered a treatment that has been sought, but not actually practicing mindfulness.

An unintended example: How not to recruit cancer patients for a psychological intervention trial

Randomized-controlled-trials-designsSometimes PhD students just can’t be dissuaded from undertaking an evaluation of a psychotherapy. I was a member of a PhD committee of a student who at least produced a valuable paper concerning how not to recruit cancer patients for a trial evaluating problem-solving therapy, even though the project fell far short of conducting an adequately powered study.

The PhD student was aware that  claims of effectiveness of problem-solving therapy reported in in the prestigious Journal of Consulting and Clinical Psychology were exaggerated. The developer of problem-solving therapy for cancer patients (and current JCCP Editor) claimed  a huge effect size – 3.8 if only the patient were involved in treatment and an even better 4.4 if the patient had an opportunity to involve a relative or friend as well. Effect sizes for this trial has subsequently had to be excluded from at least meta-analyses as an extreme outlier (1,2,3,4).

The student adopted the much more conservative assumption that a moderate effect size of .6 would be obtained in comparison with a waitlist control. You can use G*Power to see that 50 patients would be needed per group, 60 if allowance is made for dropouts.

Such a basically inert control group, of course, has a greater likelihood of seeming to demonstrate a treatment is effective than when the comparison is another active treatment. Of course, such a control group also has the problem of not allowing a determination if it was the active ingredient of the treatment that made the difference, or just the attention, positive expectations, and support that were not available in the waitlist control group.

But PhD students should have the same option as their advisors to contribute another comparison between an active treatment and a waitlist control to the literature, even if it does not advance our knowledge of psychotherapy. They can take the same low road to a successful career that so many others have traveled.

This particular student was determined to make a different contribution to the literature. Notoriously, studies of psychotherapy with cancer patients often fail to recruit samples that are distressed enough to register any effect. The typical breast cancer patient, for instance, who seeks to enroll in a psychotherapy or support group trial does not have clinically significant distress. The prevalence of positive effects claimed in the literature for interventions with cancer patients in published studies likely represents a confirmation bias.

The student wanted to address this issue by limiting patients whom she enrolled in the study to those with clinically significant distress. Enlisting colleagues, she set up screening of consecutive cancer patients in oncology units of local hospitals. Patients were first screened for self-reported distress, and, if they were distressed, whether they were interested in services. Those who met both criteria were then re-contacted to see if that be willing to participate in a psychological intervention study, without the intervention being identified. As I reported in the previous blog post:

  • Combining results of  the two screenings, 423 of 970 patients reported distress, of whom 215 patients indicated need for services.
  • Only 36 (4% of 970) patients consented to trial participation.
  • We calculated that 27 patients needed to be screened to recruit a single patient, with 17 hours of time required for each patient recruited.
  • 41% (n= 87) of 215 distressed patients with a need for services indicated that they had no need for psychosocial services, mainly because they felt better or thought that their problems would disappear naturally.
  • Finally, 36 patients were eligible and willing to be randomized, representing 17% of 215 distressed patients with a need for services.
  • This represents 8% of all 423 distressed patients, and 4% of 970 screened patients.

So, the PhD student’s heroic effort did not yield the sample size that she anticipated. But she ended up making a valuable contribution to the literature that challenges some of the basic assumptions that were being made about how cancer patients in psychotherapy research- that all or most were distressed. She also ended up producing some valuable evidence that the minority of cancer patients who report psychological distress are not necessarily interested in psychological interventions.

Fortunately, she had been prepared to collect systematic data about these research questions, not just scramble within a collapsing effort at a clinical trial.

Becoming a research parasite as an alternative to PhD students attempting an under-resourced study of their own

research parasite awardPsychotherapy trials represent an enormous investment of resources, not only the public funding that is often provided for them,be a research parasite but in the time, inconvenience, and exposure to ineffective treatments experienced by patients who participate in the trials. Increasingly, funding agencies require that investigators who get money to do a psychotherapy study some point make their data available for others to use.  The 14 prestigious medical journals whose editors make up the International Committee of Medical Journal Editors (ICMJE) each published in earlier in 2016 a declaration that:

there is an ethical obligation to responsibly share data generated by interventional clinical trials because participants have put themselves at risk.

These statements proposed that as a condition for publishing a clinical trial, investigators would be required to share with others appropriately de-identified data not later than six months after publication. Further, the statements proposed that investigators describe their plans for sharing data in the registration of trials.

Of course, a proposal is only exactly that, a proposal, and these requirements were intended to take effect only after the document is circulated and ratified. The incomplete and inconsistent adoption of previous proposals for registering of  trials in advance and investigators making declarations of conflicts of interest do not encourage a lot of enthusiasm that we will see uniform implementation of this bold proposal anytime soon.

Some editors of medical journals are already expressing alarmover the prospect of data sharing becoming required. The editors of New England Journal of Medicine were lambasted in social media for their raising worries about “research parasites”  exploiting the availability of data:

a new class of research person will emerge — people who had nothing to do with the design and execution of the study but use another group’s data for their own ends, possibly stealing from the research productivity planned by the data gatherers, or even use the data to try to disprove what the original investigators had posited. There is concern among some front-line researchers that the system will be taken over by what some researchers have characterized as “research parasites.”

 Richard Lehman’s  Journal Review at the BMJ ‘s blog delivered a brilliant sarcastic response to these concerns that concludes:

I think we need all the data parasites we can get, as well as symbionts and all sorts of other creatures which this ill-chosen metaphor can’t encompass. What this piece really shows, in my opinion, is how far the authors are from understanding and supporting the true opportunities of clinical data sharing.

However, lost in all the outrage that The New England Journal of Medicine editorial generated was a more conciliatory proposal at the end:

How would data sharing work best? We think it should happen symbiotically, not parasitically. Start with a novel idea, one that is not an obvious extension of the reported work. Second, identify potential collaborators whose collected data may be useful in assessing the hypothesis and propose a collaboration. Third, work together to test the new hypothesis. Fourth, report the new findings with relevant coauthorship to acknowledge both the group that proposed the new idea and the investigative group that accrued the data that allowed it to be tested. What is learned may be beautiful even when seen from close up.

The PLOS family of journals has gone on record as requiring that all data for papers published in their journals be publicly available without restriction.A February 24, 2014 PLOS’ New Data Policy: Public Access to Data  declared:

In an effort to increase access to this data, we are now revising our data-sharing policy for all PLOS journals: authors must make all data publicly available, without restriction, immediately upon publication of the article. Beginning March 3rd, 2014, all authors who submit to a PLOS journal will be asked to provide a Data Availability Statement, describing where and how others can access each dataset that underlies the findings. This Data Availability Statement will be published on the first page of each article.

Many of us are aware of the difficulties in achieving this lofty goal. I am holding my breath and turning blue, waiting for some specific data.

The BMJ has expanded their previous requirements for data being available:

Loder E, Groves T. The BMJ requires data sharing on request for all trials. BMJ. 2015 May 7;350:h2373.

The movement to make data from clinical trials widely accessible has achieved enormous success, and it is now time for medical journals to play their part. From 1 July The BMJ will extend its requirements for data sharing to apply to all submitted clinical trials, not just those that test drugs or devices. The data transparency revolution is gathering pace.

I am no longer heading dissertation committees after one that I am currently supervising is completed. But if any PhD students asked my advice about a dissertation project concerning psychotherapy, I would strongly encourage them to enlist their advisor to identify and help them negotiate access to a data set appropriate to the research questions they want to investigate.

Most well-resourced psychotherapy trials have unpublished data concerning how they were implemented, with what bias and with which patient groups ending up underrepresented or inadequately exposed to the intensity of treatment presumed to be needed for benefit. A story awaits to be told. The data available from a published trial are usually much more adequate than then any graduate student could collect with the limited resources available for a dissertation project.

I look forward to the day when such data is put into a repository where anyone can access it.

until youre done In this blog post I have argued that PhD students should not take on responsibility for developing and testing a new psychotherapy for their dissertation project. I think that using data from existing published trials is a much better alternative. However, PhD students may currently find it difficult, but certainly not impossible to get appropriate data sets. I certainly am not recruiting them to be front-line infantry in advancing the cause of routine data sharing. But they can make an effort to obtain such data and they deserve all support they can get from their dissertation committees in obtaining data sets and in recognizing when realistically that data are not being made available, even when the data have been promised to be available as a condition for publishing. Advisors, please request the data from published trials for your PhD students and protect them from the heartache of trying to collect such data themselves.

 

Creating the illusion that mindfulness improves the survival of cancer patients

  • A demonstration of just how unreliable investigators’ reports of mindfulness studies can be.
  • Exaggerations of efficacy combined with self-contradiction in the mindfulness literature pose problems for any sense being made of the available evidence by patients, clinicians, and those having responsibility for clinical and public policy decisions.

Despite thousands of studies, mindfulness-based stress reduction (MBSR) and related meditation approaches have not yet been shown  to be more efficacious than other active treatments for reducing stress. Nonetheless many cancer patients seek MBSR or mindfulness-based cancer recovery (MBCR) believing that they are improving their immune system and are on their way to a better outcome in “fighting” their cancer.

UVa Cancer Center
UVa Cancer Center

This unproven claim leads many cancer patients to integrative cancer centers. Once patients begin receiving treatment at these centers, they are offered a variety of other services that can be expensive, despite being unproven or having been proven ineffective. Services provided by integrative cancers treatments can discourage patients from seeking conventional treatments that are more effective, but that come with serious side effects and disfigurement. Moreover, integrative treatments give false hope to patients who would otherwise accept the limits of treatments for cancer and come to terms with their own mortality. And integrative treatments can lead to patients blaming themselves when they do not benefit.

Mindfulness studies keep being added to the literature, often in quality journals, that cultivate these illusions of vulnerable cancer patients. This psychoneuroimmunology(PNI) literature is self-perpetuating in its false claims, exaggerations, and spin. The literature ignores some basic findings:

  1. Psychotherapy and support groups have not been shown to improve the survival of cancer patients.
  2. The contribution of stress to the onset, progression, and outcome of cancer is likely to be minimal, if at all.
  3. Effects of psychological interventions like MBSR/MBCR on the immune system are weak or nonexistent, and the clinical significance of any effects is not established.

Evidence-based oncologists and endocrinologists would not take seriously the claims regularly appearing in the PNI literature. Such clinician-scientists would find bizarre many of the supposed mechanisms by which MBCR supposedly affects cancer. Yet, investigators create the illusion of accumulating evidence, undaunted by negative findings and the lack of plausible mechanisms by which MBCR could conceivably influence basic disease processes in cancer.

This blog post debunks a study by one of the leading proponents of MBCR for cancer patients, showing how exaggerated and outright false claims are created and amplified across publications.

Responsible scientists and health care providers should dispel myths that patients may have about the effectiveness of psychosocial treatments in extending life. But in the absence of responsible professionals speaking out, patients can be intimidated by how these studies are headlined in the popular media, particularly when they believe that they are dealing with expert opinion based on peer-reviewed studies.

Mindfulness-based cancer recovery (MBCR)

The primary report for study was published in the prestigious Journal of Clinical Oncology and is available as a downloadable PDF 

Carlson LE, Doll R, Stephen J, Faris P, Tamagawa R, Drysdale E, Speca M. Randomized controlled trial of mindfulness-based cancer recovery versus supportive expressive group therapy for distressed survivors of breast cancer (MINDSET).  Journal of Clinical Oncology. 2013 Aug 5:JCO-2012.

The authors compared the efficacy of what they describe as“two empirically supported group interventions to help distressed survivors of breast cancer”: mindfulness-based cancer recovery (MBCR) and supportive-expressive group therapy (SET). Each of these active treatments was delivered in 8 weekly 90 minute sessions plus a six-hour workshop. A six-hour, one day didactic seminar served as the comparison/control condition.

The 271 participants were Stage I, II, or III breast patients who had completed all cancer treatment a mean of two years ago. Patients also had to meet a minimal level of distress and not have a psychiatric diagnosis.  Use of psychotropic medication was not an exclusion, because of the high prevalence of antidepressants and anxiolytics in this population.

One hundred thirteen patients were randomized to MBSR, 104 to SET, and 54 to the didactic seminar control group.

A full range of self-measures was collected, along with saliva samples at four times (awakening, noon,  5 PM, and bedtime) over three days. The trial registration for this study is basically useless. It is lacking in basic detail.  Rather than declaring one or maybe two outcomes as primary, the authors specify broad classes – mood, stress, post-traumatic growth, social support, quality of life, spirituality and cortisol levels (stress hormone). Yet,

a later report  states that ”The sample size estimate was based on the primary outcome measure (POM TMD)” – Profile of Mood Total Mood score. The saliva collection was geared to assessing cortisol, although in such studies saliva can provide a full range of biological variables, including immune function.

Why bring up the lack of registration and multiple outcome measures?

The combination of a vague trial registration and multiple outcome measures allows investigators considerable flexibility in which outcome they pick. They can wait to make a choice until after results are known, but that is considered a questionable research practice. The collection of saliva was obviously geared to assessing saliva cortisol. However, a recent comprehensive review  of salivary diurnal cortisol as an outcome measure at least three parameters (the cortisol awakening response, diurnal slope an area under the curve), each reflecting different aspects of hypothalamus pituitary adrenal) HPA axis function.

So, the authors have a lot of options from which to choose data points and analyses best suggesting that MBCR is effective.

Results

Cortisol-levels-400x210Modest effects on POMS TMS disappeared in corrected pairwise comparisons between MBSR in SET. So, according to the evidence presented, mood was not improved.

Baseline cortisol data were only available for 242 patients, and only 172 had data for post intervention slopes. Uncorrected group differences in cortisol slope across the day are not reported. However, when cancer severity number of cigarettes smoked per day, and sleep quality were entered as control variables, a group X time difference was found (p< .009).

We should beware of studies that do not present uncorrected group differences, but depend on only data adjusted for covariates, the appropriateness of which is not established.

But going further, there was no difference between MBCR and SEM. Actually, any difference between these two groups and the control with due to an unexpected increase in the control group slope while the patients in the MBCR and SEM remained unchanged. I can’t see how this would have been predicted. The assumption guiding the study had been that cortisol slope should decrease one or both of the active intervention groups.

The authors searched for more positive findings from cortisol and found:

There were no significant group x time interaction effects for cortisol concentrations at any single collection point, but a time x group contrast between MBCR and SMS was significant for bedtime cortisol concentrations (P =.044; Table 3), which were elevated after SMS (mean change, 0.11) but slightly decreased after MBCR (mean change,=0.02; Fig 2D).

These are weak findings revealed by a post hoc search of a number of different cortisol measures. Aside from the analysis been post-hoc,  I would not place much confidence in a cherry-picked p = .044.

How the authors discuss the results

 Ignoring the null results for the primary measure, the Profile of Mood States Total Score (POM TS), the authors jump to secondary outcomes to proclaim the greater effectiveness of MBCR:

As predicted, MBCR emerged as superior for decreasing symptoms of stress and also for improving overall quality of life and social support in these women, even though we hypothesized that SET might be superior on social support. Improvements were clinically meaningful and similar to those reported in our previous work with mixed groups of patients with cancer.

Keep in mind the disappointing result for cortisol profiles when reading their closing claims for “significantly altered” cortisol:

Cortisol profiles were significantly altered after program completion. Participants in both MBCR and SET maintained the initial steepness of cortisol slopes, whereas SMS participants evidenced increasingly flatter diurnal cortisol slopes, with a medium between-group effect size. Hence, the two interventions buffered unfavorable biologic changes that may occur without active psychosocial intervention. Because abnormal or flattened cortisol profiles have been related to both poorer psychological functioning and shorter survival time in breast,16,17,45,46 lung,47 and renal cell48 carcinoma, this finding may point to the potential for these psychosocial interventions to improve biologic processes related to both patient-reported outcomes and more objective indices. More work is needed to fully understand the clinical meaning of these parameters in primary breast cancer.

The authors set out to demonstrate this psychological interventions decreased cortisol slopes and found no evidence that they did. However, they seized on the finding of increasingly flatter cortisol slopes in the control group. But all these breast cancer patients are receiving MBCR and SET two years after their cancer treatment ended. For most patients, distress levels have receded by then to what they were before cancer was detected. One has to ask the authors if they are taking seriously this continuing decline in cortisol slopes, where are cortisol levels heading?  And when did the decline start?

I attach no credibility to the authors’ claims unless they provide us with an understanding of how they occurred. Do the authors assume they have an odd group of patients who have been declining since diagnosis or maybe since the end of active cancer treatment, but have somehow ended up at the same level of cortisol as the other patients in the sample? There was, you know, random assignment and, there were no baseline differences at the start of this study.

The attempt to relate their findings to shorter survival time in a variety of cancers is dodgy and irresponsible. Their overview of the literature is highly selective, depends on small samples, and there is no evidence that the alleged flattened cortisol profiles are causes rather than being an effect of disease parameters associated with shorter survival.

The authors have not demonstrated an effect of their psychological interventions on survival. No previous study ever has.

Interestingly, a classic small study by Spiegel prompted a whole line of research in which an effect of psychological intervention on survival was sought. However, a careful look at the graphs in his original study reveals that the survival curves for the patients receiving the intervention approximated with other patients with advanced breast cancer in the larger community in the absence of intervention. Compared to the large population from which they were drawn, the patients receiving the intervention in Spiegel’s study were no better off.

survival curve-page-0In the contrast, there were unexplainable deaths in Spiegel’s  control group that generated the illusion that his intervention was increasing survival. Given how small his control group was (39 patients at the outset), it only took the sudden death of four patients in the control group to create an effect where previously there was none. So, it is not that psychotherapy extended survival,  but that a small cluster of patients in the control group died suddenly, years after randomization. Go figure, but keep in mind that the study was never designed to test the effects of psychological intervention on survival. That hypothesis was generated after data were available and Spiegel claimed surprise that they were positive findings.

Spiegel himself has never been able to replicate this finding. You can read more about this study here.

From Hilda Bastian
From Hilda Bastian

The present authors did not identify survival has a primary outcome for the trial, nor did they assess it. They are essentially depending on spun data that assumes cortisol slope not just as a biological variable, but a surrogate for survival. See a blog post by Hilda Bastian’s Statistically funny: Biomarkers Unlimited: Accept Only OUR Substitutes!  for an explanation of why this is sheer folly. Too many promising medical treatments for cancer have been accepted as efficacious on the basis of surrogate outcomes, only to be later shown to have no effect on survival. But these psychological treatments are not even in the running.

This is the kind of nonsense that encourages cancer patients to continue with the false hope that mindfulness-based treatment will extend their lives.

1681869-slide-aforanimation-sitonmyface-1The fish gets bigger with each telling.

A follow up paper  makes stronger claims and makes new claims of telomere length,  the clinical implications of which the authors ultimately concede they don’t understand.

Carlson LE, Beattie TL, Giese‐Davis J, Faris P, Tamagawa R, Fick LJ, Degelman ES, Speca M. Mindfulness‐based cancer recovery and supportive‐expressive therapy maintain telomere length relative to controls in distressed breast cancer survivors. Cancer. 2015 Feb 1;121(3):476-84.

The authors opening summary of their previously reported results we have been discussing:

We recently reported primary outcomes of the MINDSET trial, which compared 2 empirically supported psychosocial group interventions, mindfulness-based cancer recovery (MBCR) and supportive-expressive group therapy (SET), with a minimal-intervention control condition on mood, stress symptoms, quality of life, social support, and diurnal salivary cortisol in distressed breast cancer survivors.[4] Although MBCR participation resulted in the most psychosocial benefit, including improvements across a range of psychosocial outcomes, both MBCR and SET resulted in healthier cortisol profiles over time compared with the control condition.

Endocrinologists would scratch their heads and laugh at the claim that intervention resulted in “healthier cortisol profiles.” There is a wide range of cortisol values in the general population, and these are well within the normal range. The idea that they are somehow “healthier” is as bogus as claims made for super foods and supplements. You have to ask, “healthier” in what sense?

In this secondary analysis of MINDSET trial data, we collected and stored blood samples taken from a subset of women to further investigate the effects of these interventions on potentially important biomarkers. Telomeres are specialized nucleoprotein complexes that form the protective ends of linear chromosomes and provide genomic stability through several mechanisms.

The authors justify the study with speculations that stop just short of claiming their intervention increased survival:

Telomere dysfunction and the loss of telomere integrity may result in DNA damage or cell death; when a critically short telomere length (TL) is reached, cells enter senescence and have reduced viability, and chromosomal fusions appear.[6] Shorter TL has been implicated in several disease states, including cardiovascular disease, diabetes, dyskeritosis congenita, aplastic anemia, and idiopathic pulmonary fibrosis.[7] Shorter TL also was found to be predictive of earlier mortality in patients with chronic lymphocytic leukemia,[8] promyelocytic leukemia,[9] and breast cancer.[10-12] However, the relationships between TL and the clinical or pathological features of tumors are still not clearly understood.[13].

They waffle some more and then acknowledge there are few relevant data concerning cancer:

Telomere dysfunction and the loss of telomere integrity may result in DNA damage or cell death; when a critically short telomere length (TL) is reached, cells enter senescence and have reduced viability, and chromosomal fusions appear.[6] Shorter TL has been implicated in several disease states, including cardiovascular disease, diabetes, dyskeritosis congenita, aplastic anemia, and idiopathic pulmonary fibrosis.[7] Shorter TL also was found to be predictive of earlier mortality in patients with chronic lymphocytic leukemia,[8] promyelocytic leukemia,[9] and breast cancer.[10-12] However, the relationships between TL and the clinical or pathological features of tumors are still not clearly understood.[13].

Too small a sample to find anything clinically significant and generalizable

Correlational studies of telomere length and disease require very large samples. These epidemiologic findings in no way encourage anticipating finding effects in a modest sized trial of a psychological intervention. Moreover, significant results from smaller studies exaggerate associations because they have to be larger to be statistically significant.  They not be expected to replicate in a larger study. The authors’ sample has shrunk considerably from recruitment and randomization to a sample of women provided two blood samples with which they hope to find differences among two interventions and one control group.

Due to the availability of resources, blood samples were only collected in Calgary. Of the 128 women in Calgary, 5 declined to donate their blood. Thirty-one women provided their blood only at the preintervention time period; therefore, the current study included 92 women who donated a blood sample before and after the intervention.

Not surprisingly, no differences between groups were found, but that inspires some creativity in analysis.

The results of ANCOVA demonstrated no statistical evidence of differences in postintervention TL between the MBCR and SET interventions after adjusting the impact of the preintervention log10 T/S ratios. The mean difference was −0.12 (95% confidence interval [95% CI], −0.74 to 0.50). Because the 2 interventions shared similar nonspecific components and no significant differences emerged in their baseline-adjusted postintervention T/S ratios, the 2 intervention groups were subsequently combined to allow greater power for detecting any effects on TL related to participation in a psychosocial intervention compared with the control condition.

The authors initially claimed that MBCR and SET were so different that an expensive large scale RCT was justified. Earlier in the present paper they claimed MBCR was superior. But now they are claiming there is so little difference between  treatments that a post hoc combining is justified to see if null findings can be overturned.

Their tortured post hoc analyses revealed a tiny effect that they fail to acknowledge was nonsignificat – confidence intervals (-0.01 to 1.35)  include 0:

After adjustment for the baseline log10 T/S ratio, there was a statistical trend toward a difference in posttreatment log10 T/S ratios between treatment and control subjects (statistics shown in Table 2). The adjusted mean difference was 0.67 (95% CI, -0.01 to 1.35). The effect size of g2 was 0.043 (small to medium).

There was no association between psychological outcomes and telomere length. Yet differences would be expected if interventions targeting psychological variables somehow influenced telomere length.

Nonetheless, the authors concluded they had a pattern in the results of the primary and secondary studies encouraging more research:

Together, these changes suggest an effect of the interventions on potentially important biomarkers of psychosocial stress. Given the increasingly well-documented association between TL and cancer initiation46 and survival,47 this finding adds to the literature supporting the potential for stress-reducing interventions to impact important disease-regulating processes and ultimately disease outcome.

They end with a call for bigger, more expensive studies, even if they cannot understand what is going on (or for that matter, whether anything of interest occurred in their study):

Future investigators should power studies of intervention effects on TL and telomerase as primary outcomes, and follow participants over time to better understand the clinical implications of group differences. The interpretation of any changes in TL in patients with breast cancer is difficult. One study that analyzed TL in breast tumor tissue found no relations between TL and any clinical or pathological features or disease or survival outcomes,13 whereas other studies have shown that TL was related to breast cancer risk46,51 and survival.10,46,47 Although interpretation remains difficult,the results of the current study nonetheless provide provocative new data that suggest it is possible to influence TL in cancer survivors through the use of psychosocial interventions involving group support, emotional expression, stress reduction, and mindfulness meditation.

This is not serious research. At the outset, the authors had to know that the sample was much too small and there been too much nonrandom attrition to make robust and generalizable conclusions concerning effects on telomere length. And the authors knew ahead of time, they had no idea how they would interpret such effect. But they didn’t find them. They delivered an intervention, administered questionnaires, took spit and blood samples, but this is not “research” in which they were willing to concede hypotheses were confirmed, this is an experimercial for mindfulness programs.

exaggeration-300x290But the power of MBCR gets even greater with yet another telling

A recent review:

Carlson LE. Mindfulness‐based interventions for coping with cancer. Annals of the New York Academy of Sciences. 2016 Mar

One of the authors of the two articles we have been discussing uses them as the main basis for even stronger claims and about MBCR specifically.

Our adaptation, mindfulness-based cancer recovery (MBCR), has resulted in improvements across a range of psychological and biological outcomes, including cortisol slopes, blood pressure, and telomere length, in various groups of cancer survivors.

Wow! Specifically,

Overall, women in the MBCR group showed more improvement on stress symptoms compared with women in both the SET and control groups, on QOL compared with the control group, and in social support compared with the SET group,[28] but both active-intervention groups’ cortisol slopes (a marker of stress responding) were maintained over time relative to the control group, whose cortisol slopes became flatter. Steeper slopes are generally considered to be healthier. The two intervention groups also maintained their telomere length, a potentially important marker of cell aging, over time compared to controls,

But wait! The superiority of MBCR gets even better with a follow-up study.

The publication of long term follow up data become the occasion for describing the superiority of MBCR over SET as ever greater.

Carlson LE, Tamagawa R, Stephen J, Drysdale E, Zhong L, Speca M. Randomized‐controlled trial of mindfulness‐based cancer recovery versus supportive expressive group therapy among distressed breast cancer survivors (MINDSET): long‐term follow‐up results. Psycho‐Oncology. 2016 Jan 1.

 The abstract describes the outcomes at the end of the intervention:

Immediately following the intervention, women in MBCR reported greater reduction in mood disturbance (primarily fatigue, anxiety and confusion) and stress symptoms including tension, sympathetic arousal and cognitive symptoms than those in SET. They also reported increased emotional and functional quality of life, emotional, affective and positive social support, spirituality (feelings of peace and meaning in life) and post-traumatic growth (appreciation for life and ability to see new possibilities) relative to those in SET, who also improved to a lesser degree on many outcomes.

A search for “cortisol” in this report finds it is never mentioned.

The methods section clarifies that the 54 women in the seminar control group were offered randomization to the two active treatments and 35 accepted, with 21 going to MBCR and 14 to SET. However, 8 of the women newly assigned to MBCR and 9 of the women newly assigned to SET did not provide post-intervention data.  The authors nonetheless used two-level piecewise hierarchical linear modelling (HLM) with random intercepts for intent-to-treat analyses for the full sample. The authors acknowledge a high attrition rate of over half of the patients being lost to follow up, but argue these hierarchical analyses were a solution. While this is often done, the analyses assumes attrition is random and validity is vulnerable to such high rates of attrition. I don’t know why a reviewer did not object to the analyses or the strong conclusions drawn from them.

Recognize what is being done here: the authors are including a small amount of new data in analyses, but with so much attrition by the end of treatment  that analyses depend more on estimating from data available from a minority of patients to what the authors claim would be obtained if the full sample were involved. This is statistically dodgy, but apparently acceptable to the stats editor of this journal. What the authors did is not considered fraud, but it is making up data.

The follow up study concludes:

In sum, these results represent the first demonstration in a comparative effectiveness approach that MBCR is superior to another active intervention, SET, which also showed lesser benefit to distressed survivors of breast cancer. Our previous report also showed that MBCR was superior to a minimal intervention control condition pre-intervention to post-intervention. Benefits were accrued across outcomes measuring stress, mood, quality of life and PTG, painting a picture of women who were more able to cope with cancer survivorship and to fully embrace and enjoy life.

I pity the poor detached investigator attempting to use these data in a meta-analysis. Do they go with the original, essentially null results, or do they rely on these voodoo statistics that post-hoc give a better picture. They would have to write to the authors anyway, because on corrected results are presented in the paper.

This is not science, it is promotion of a treatment by enthusiastic proponents who are strongly committed to demonstrating that the treatment is superior to alternatives, in defiance of contradictory data they have generated.

Terribly disappointing, but this effort is actually better than much of the studies of mindfulness for cancer patients. It is a randomized trial, and started with a reasonably large sample, even if it has substantial attrition – i.e., most patients were lost to follow-up.

For those you who have actually read this longread blog post from start to finish, would you have expected this kind of background if you’d only stumbled upon the authors’ glowing praise of their own work in the prestigious Annals of New York Academy of Sciences? I don’t think so.

Dammit! It shouldn’t be so hard to figure out what went on studies. We should be able to depend on authors to provide more transparent, consistent reports of the results they obtain. While mindfulness research has no monopoly on such contrary practices, it is exceptionally rich with exaggerated and even false claims and suppression of evidence to the contrary. Consumers be very skeptical of what they read!

Let’s get more independent re-evaluations of the claims made by promoters of mindfulness by those who don’t profit professionally or financially from exaggerating benefits. And please, clinicians, start dispelling the myths of cancer patients who think that they are obtaining effects on their disease from practicing mindfulness.

For further discussion, see Mindfulness-based stress reduction for improving sleep among cancer patients: A disappointing look.