Headspace mindfulness training app no better than a fake mindfulness procedure for improving critical thinking, open-mindedness, and well-being.

The Headspace app increased users’ critical thinking and being open-minded. So did practicing a shame mindfulness procedure- participants simply sat with their eyes closed, but thought they were meditating.

mind the brain logo

The Headspace app increased users’ critical thinking and open-mindedness. So did practicing a sham mindfulness procedure. Participants simply sat with their eyes closed, but thought they were meditating.

cat_ dreamstime_164683 (300 x 225)Results call into question claims about Headspace  coming from other studies that did not have such a credible, active control group comparison.

Results also call into question the widespread use of standardized self-report measures of mindfulness to establish whether someone is in the state of mindfulness. These measures don’t distinguish between the practice of standard versus fake mindfulness.

Results can be seen as further evidence that practicing mindfulness depends on nonspecific factors (AKA placebo), rather than any active, distinctive ingredient.

Hopefully this study will prompt better studies evaluating the Headspace App, as well as evaluations of mindfulness training more generally, using credible active treatments, rather than no treatment or waitlist controls.

Maybe it is time for a moratorium on trials of mindfulness without such an active control or at least a tempering of claims based on poorly controlled  trials.

This study points to the need for development of more psychometrically sophisticated measures of mindfulness that are not so vulnerable to experiment expectations and demand characteristics.

Until the accumulation of better studies with better measures, claims about the effects of practicing mindfulness ought to be recognized as based on relatively weak evidence.

The study

Noone, C & Hogan,M. Randomised active-controlled trial of effects of online mindfulness intervention on executive control, critical thinking and key thinking dispositionsBMC Psychology, 2018

Trial registration

The study was initially registered in the AEA Social Science Registry before the recruitment was initiated (RCT ID: AEARCTR-0000756; 14/11/2015) and retrospectively registered in the ISRCTN registry (RCT ID: ISRCTN16588423) in line with requirements for publishing the study protocol.

Excerpts from the Abstract

The aim of this study was…investigating the effects of an online mindfulness intervention on executive function, critical thinking skills, and associated thinking dispositions.

Method

Participants recruited from a university were randomly allocated, following screening, to either a mindfulness meditation group or a sham meditation group. Both the researchers and the participants were blind to group allocation. The intervention content for both groups was delivered through the Headspace online application, an application which provides guided meditations to users.

And

Primary outcome measures assessed mindfulness, executive functioning, critical thinking, actively open-minded thinking, and need for cognition. Secondary outcome measures assessed wellbeing, positive and negative affect, and real-world outcomes.

Results

Significant increases in mindfulness dispositions and critical thinking scores were observed in both the mindfulness meditation and sham meditation groups. However, no significant effects of group allocation were observed for either primary or secondary measures. Furthermore, mediation analyses testing the indirect effect of group allocation through executive functioning performance did not reveal a significant result and moderation analyses showed that the effect of the intervention did not depend on baseline levels of the key thinking dispositions, actively open-minded thinking, and need for cognition.

The authors conclude

While further research is warranted, claims regarding the benefits of mindfulness practice for critical thinking should be tempered in the meantime.

Headscape Be used on an iPhone

The active control condition

The sham treatment control condition was embarrassingly straightforward and simple. But as we will see, participants found it credible.

This condition presented the participants with guided breathing exercises. Each session began by inviting the participants to sit with their eyes closed. These exercises were referred to as meditation but participants were not given guidance on how to control their awareness of their body or breath. This approach was designed to control for the effects of expectations surrounding mindfulness and physiological relaxation to ensure that the effect size could be attributed to mindfulness practice specifically. This content was also delivered by Andy Puddicombe and was developed based on previous work by Zeidan and colleagues [55, 57, 58].

What can we conclude about the standard self-report measures of the state of mindfulness?

The study used the Five Facet Mindfulness Questionnaire, which is widely used to assess whether people are in a state of mindfulness. It has been cited almost 4000 times.

Participants assigned to the mindfulness condition had significant changes for all five facets from baseline to follow up: observing, non-reactivity, non-judgment, acting with awareness, and describing. In the absence of a comparison with change in the sham mindfulness group, these pre-post results would seem to suggest that the measure was sensitive to whether participants had practiced mindfulness. However, there were no differences from the changes observed for the participants assigned to mindfulness and those which were simply asked to sit with their eyes closed.

I asked Chris Noone about the questionnaires his group used to assess mindfulness:

The participants genuinely thought they were meditating in the sham condition so I think both non-specific and demand characteristics were roughly equivalent across both groups. I’m also skeptical regarding the ability of the Five-Facet Mindfulness Questionnaire (or any mindfulness questionnaire for that matter) to capture anything other than “perceived mindfulness”. The items used in these questionnaires feature similar content to the scripts used by the people delivering the mindfulness (and sham) guided meditations. The improvement in critical thinking across both groups is just a mix of learning across a semester and habituation to the task (as the same problems were posed at both measurements).

What I like about this trial

The trial provides a critical test of a key claim for mindfulness:

Mindfulness should facilitate critical thinking in higher-education, based on early Buddhist conceptualizations of mindfulness as clarity of thought.

The trial was registered before recruitment and departures from protocol were noted.

Sample size was determined by power analysis.

The study had a closely matched, active control condition, a sham mindfulness treatment.

The credibility and equivalence of this sham condition versus the active treatment under study was repeatedly assessed.

“Manipulation checks were carried out to assess intervention acceptability, technology acceptance and meditation quality 2 weeks after baseline and 4 weeks after baseline.”

The study tested some a priori hypotheses about mediators and moderation:

Analyses were intention to treat.

 How the study conflicts with past studies

Previous studies claimed to show positive effects of mindfulness on aspects of executive functioning [25 and  26]

How the contradiction of past studies by these results is resolved

 “There are many studies using guided meditations similar to those in our mindfulness meditation condition, delivered through smartphone applications [49, 50, 52, 90, 91], websites [92, 93, 94, 95, 96, 97] and CDs [98, 99], which show effects on measures of outcomes reliably associated with increases in mindfulness such as depression, anxiety, stress, wellbeing and compassion. There are two things to note about these studies – they tend not to include a measure of dispositional mindfulness (e.g. only 4% of all mindfulness intervention studies reviewed in a recent meta-analysis include such measures at baseline and follow-up; [54]) and they usually employ a weak form of control group such as a no-treatment control or waitlist control [54]. Therefore, even when change in mindfulness is assessed in mindfulness meditation intervention studies, it is usually overestimated and this must be borne in mind when comparing the results of this study with those of previous studies. This combined with generally only moderate correlations with behavioural outcomes [54] suggests that when mindfulness interventions are effective, dispositional measures do not fully capture what has changed.”

The broader take away messages

“Our results show that, for most outcomes, there were significant changes from baseline to follow-up but none which can be specifically attributed to the practice of mindfulness.’

This creative use of a sham mindfulness control condition is a breakthrough that should be widely followed. First, it allowed a fair test of whether mindfulness is any better than another active, credible treatment. Second, because the active treatment was a sham, results provide a challenge to the notion that apparent effects of mindfulness on critical thinking are anything more than a placebo effect.

The Headspace App is enormously popular and successful, based on claims about what benefits its use will provide. Some of these claims may need to be tempered, not only in terms of critical thinking, but effects on well-being.

The Headspace App platform lends itself to such critical evaluations with respect to a sham treatment with a degree of standardization that is not readily possible with face-to-face mindfulness training. This opportunity should be exploited further with other active control groups constructed on the basis of specific hypotheses.

There is far too much research on the practice of mindfulness being done that does not advance understanding of what works or how it works. We need a lot fewer studies, and more with adequate control/comparison groups.

Perhaps we should have a moratorium on evaluations of mindfulness without adequate control groups.

Perhaps articles being aimed at audiences making enthusiastic claims for the benefits of mindfulness should routinely note whether these claims are based on adequately controlled studies. Most are not.

Flawed meta-analysis reveals just how limited the evidence is mapping meditation into specific regions of the brain

The article put meaningless, but reassuring effect sizes into the literature where these numbers will be widely and uncritically cited.

mind the brain logo

“The only totally incontrovertible conclusion is that much work remains to be done…”.

lit up brain not in telegraph article PNG

Authors of a systematic review and meta-analysis of functional neuroanatomical studies (fMRI and PET) of meditation were exceptionally frank in acknowledging problems relating the practice of meditation to differences in specific regions of the brain. However, they did not adequately deal with problems hiding in plain sight. These problems should have discouraged integration of this literature into a meta-analysis and the authors’ expressing the strength of the association between meditation and the brain in terms of a small set of moderate effect sizes.

The article put meaningless, but reassuring effect sizes into the literature where these numbers will be widely and uncritically cited.

An amazing set of overly small studies with evidence that null findings are being suppressed.

Many in the multibillion mindfulness industry are naive or simply indifferent to what constitutes quality evidence. Their false confidence that “meditation changes the brain*” can be bolstered by selective quotes from this review seemingly claiming that the associations are well-established and practically significant. Readers who are more sophisticated may nonetheless be mislead by this review, unless they read beyond the abstract and with appropriate skepticism.

Read on. I suspect you will be surprised as I was about the small quantity and poor quality of the literature relating the practice of meditation to specific areas of the brain. The colored pictures of the brain widely used to illustrate discussions of meditation are premature and misleading.

As noted in another article :

Brightly coloured brain scans are a media favourite as they are both attractive to the eye and apparently easy to understand but in reality they represent some of the most complex scientific information we have. They are not maps of activity but maps of the outcome of complex statistical comparisons of blood flow that unevenly relate to actual brain function. This is a problem that scientists are painfully aware of but it is often glossed over when the results get into the press.

The article is

Fox KC, Dixon ML, Nijeboer S, Girn M, Floman JL, Lifshitz M, Ellamil M, Sedlmeier P, Christoff K. Functional neuroanatomy of meditation: A review and meta-analysis of 78 functional neuroimaging investigations. Neuroscience & Biobehavioral Reviews. 2016 Jun 30;65:208-28.

Abstract.

Keep in mind how few readers go beyond an abstract in forming an impression of what an article shows. More readers “know” what the meta analysis found solely based on their reading the abstract , relative to the fewer people who read both the article and the supplementary material).

Meditation is a family of mental practices that encompasses a wide array of techniques employing distinctive mental strategies. We systematically reviewed 78 functional neuroimaging (fMRI and PET) studies of meditation, and used activation likelihood estimation to meta-analyze 257 peak foci from 31 experiments involving 527 participants. We found reliably dissociable patterns of brain activation and deactivation for four common styles of meditation (focused attention, mantra recitation, open monitoring, and compassion/loving-kindness), and suggestive differences for three others (visualization, sense-withdrawal, and non-dual awareness practices). Overall, dissociable activation patterns are congruent with the psychological and behavioral aims of each practice. Some brain areas are recruited consistently across multiple techniques—including insula, pre/supplementary motor cortices, dorsal anterior cingulate cortex, and frontopolar cortex—but convergence is the exception rather than the rule. A preliminary effect-size meta-analysis found medium effects for both activations (d = 0.59) and deactivations (d = −0.74), suggesting potential practical significance. Our meta-analysis supports the neurophysiological dissociability of meditation practices, but also raises many methodological concerns and suggests avenues for future research.

The positive claims in the abstract

“…Found reliably dissociable patterns of brain activation and deactivation for four common styles of meditation.”

“Dissociable activation patterns are congruent with the psychological and behavioral aims of each practice.”

“Some brain areas are recruited consistently across multiple techniques”

“A preliminary effect-size meta-analysis found medium effects for both activations (d = 0.59) and deactivations (d = −0.74), suggesting potential practical significance.”

“Our meta-analysis supports the neurophysiological dissociability of meditation practices…”

 And hedges and qualifications in the abstract

“Convergence is the exception rather than the rule”

“[Our meta-analysis] also raises many methodological concerns and suggests avenues for future research.

Why was this systematic review and meta-analysis undertaken now?

A figure provided in the article showed a rapid accumulation of studies of mindfulness in the brain in the past few years, with over 100 studies now available.

However, the authors systematic search yielded “78 functional neuroimaging (fMRI and PET) studies of meditation, and used activation likelihood estimation to meta-analyze 257 peak foci from 31 experiments involving 527 participants.” About a third of the studies identified in a search provided usable data.

What did the authors want to accomplish?

Taken together, our central aims were to: (i) comprehensively review and meta-analyze the existing functional neuroimaging studies of meditation (using the meta-analytic method known as activation likelihood estimation, or ALE), and compare consistencies in brain activation and deactivation both within and across psychologically distinct meditation techniques; (ii) examine the magnitude of the effects that characterize these activation patterns, and address whether they suggest any practical significance; and (iii) articulate the various methodological challenges facing the emerging field of contemplative neuroscience (Caspi and Burleson, 2005; Thompson, 2009; Davidson, 2010; Davidson and Kaszniak, 2015), particularly with respect to functional neuroimaging studies of meditation.

Said elsewhere in the article:

Our central hypothesis was a simple one: meditation practices distinct at the psychological level (Ψ) may be accompanied by dissociable activation patterns at the neurophysiological level (Φ). Such a model describes a ‘one-to-many’ isomorphism between mind and brain: a particular psychological state or process is expected to have many neurophysiological correlates from which, ideally, a consistent pattern can be discerned (Cacioppo and Tassinary, 1990).

The assumption is meditating versus not-meditating brains should be characterized by  distinct, observable neurophysiological pattern. There should also be distinct, enduring changes in the brain in people who have been practicing meditation for some time.

I would wager that many meditation enthusiasts believe that links to specific regions are already well established. Confronted with evidence to the contrary, they would suggest that links between the experience of meditating and changes in the brain are predictable and are waiting to be found. It is that kind of confidence that leads to the significance chasing and confirmatory bias currently infecting this literature.

Types of meditation available for study

Quantitative analyses focused on four types of meditation. Additional terms of meditation did not have sufficient studies and so were examined qualitatively. Some studies of the four provided within-group effect size, whereas other studies provided between-group effect sizes.

Focused attention (7 studies)

Directing attention to one specific object (e.g., the breath or a mantra) while monitoring and disengaging from extraneous thoughts or stimuli (Harvey, 1990, Hanh, 1991, Kabat-Zinn, 2005, Lutz et al., 2008b, Wangyal and Turner, 2011).

Mantra recitation (8 studies)

Repetition of a sound, word, or sentence (spoken aloud or silently in one’s head) with the goals of calming the mind, maintaining focus, and avoiding mind-wandering.

Open monitoring (10 studies)

Bringing attention to the present moment and impartially observing all mental contents (thoughts, emotions, sensations, etc.) as they naturally arise and subside.

Loving-kindness/compassion (6 studies)

L-K involves:

Generating feelings of kindness, love, and joy toward themselves, then progressively extend these feelings to imagined loved ones, acquaintances, strangers, enemies, and eventually all living beings (Harvey, 1990, Kabat_Zinn, 2005, Lutz et al., 2008a).

Similar but not identical, compassion meditation

Takes this practice a step further: practitioners imagine the physical and/or psychological suffering of others (ranging from loved ones to all humanity) and cultivate compassionate attitudes and responses to this suffering.

In addition to these four types of meditation, three others can be identified, but so far have only limited studies of the brain: Visualization, Sense-withdrawal and Non-dual awareness practices.

A dog’s breakfast: A table of the included studies quickly reveals a meta-analysis in deep trouble

studies included

This is not a suitable collection of studies to enter into a meta-analysis with any expectation that a meaningful, generalizable effect size will be obtained.

Most studies (14) furnish only pre-post, within-group effects for mindfulness practiced by long time practitioners. Of these 14 studies, there are two outliers with 20 and 31 practitioners. Otherwise the sample size ranges from 4 to 14.

There are 11 studies furnishing between-group comparisons between experienced and novice meditators. The number of participants in the smaller cell is key for the power of between-group effect sizes, not the overall sample size. In these 11 studies, this ranged from 10 to 22.

It is well-known that one should not combine within- and between- group effect sizes in meta analysis.  Pre-/post-within-group differences capture not only the effects of the active ingredients of an intervention, but nonspecific effects of the conditions under which data are gathered, including regression to the mean. These within-group differences will typically overestimate between-group differences. Adding a  comparison group and calculating between-group differences has the potential for  controlling nonspecific effects, if the comparison condition is appropriate.

The effect sizes based on between-group differences in these studies have their own problems as estimates of the effects of meditation on the brain. Participants were not randomized to the groups, but were selected because they were already either experienced or novice meditators. Yet these two groups could differ on a lot of variables that cannot be controlled: meditation could be confounded with other lifestyle variables: sleeping better or having a better diet. There might be pre-existing differences in the brain that made it easier for the experienced meditators to have committed to long term practice. The authors acknowledge these problems late in the article, but they do so only after discussing the effect sizes they obtained as having substantive importance.

There is good reason to be skeptical that these poorly controlled between-group differences are directly comparable to whatever changes would occur in experienced meditators’ brains in the course of practicing meditation.

It has been widely appreciated that neuroimaging studies are typically grossly underpowered, and that the result is low reproducibility of findings. Having too few participants in a  study will likely yield false negatives because of an inability to achieve the effects needed to obtain significant findings. Small sample size means a stronger association is needed to be significant.

Yet, what positive findings (i.e., significant) are obtained will of necessity be larger likely to be exaggerated and not reproducible with a larger sample.

Another problem with such small cell sizes is that it cannot be assumed that effects are due to one or more participants’ differences in brain size or anatomy. One or a small subgroup of outliers could drive all significant findings in an already small sample. The assumption that statistical techniques can smooth these interindividual differences depends on having much larger samples.

It has been noted elsewhere:

Brains are different so the measure in corresponding voxels across subjects may not sample comparable information.

How did the samples get so small? Neuroanatomical studies are expensive, but why did Lazar et al (2000) have 5 rather 6 participants, or only the 4 participants that Davanger et had? Were from some participants dropped after a peeking at the data? Were studies compromised by authors not being able to recruit intended numbers of participants and having to relax entry criteria? What selection bias is there in these small samples? We just don’t know.

I am reminded of all the contentious debate that has occurred when psychoanalysts insisted on mixing uncontrolled case-series with randomized trials in the same meta-analyses of psychotherapy. My colleagues and I showed this introduces great distortion  into the literature . Undoubtedly, the same is occurring in these studies of meditation, but there is so much else wrong with this meta analysis.

The authors acknowledge that in calculating effect sizes, they combined studies measuring cerebral blood flow (positron emission tomography; PET) and blood oxygenation level (functional magnetic resonance imaging; fMRI). Furthermore, the meta-analyses combined studies that varied in the experimental tasks for which neuroanatomical data were obtained.

One problem is that even studies examining a similar form of meditation might be comparing a meditation practice to very different baseline or comparison tasks and conditions. However, collapsing across numerous different baselines or control conditions is a common (in fact, usually inevitable) practice in meta_analyses of functional neuroimaging studies…

So, there are other important sources of heterogeneity between these studies.

Generic_forest_plot
A generic forest plot. This article did not provide one.

It’s a pity that the authors did not provide a forest plot [How to read  a forest plot.]  graphically showing the confidence intervals around the effect sizes being entered into the meta-analysis.

But the authors did provide a funnel plot that I found shocking. [Recommendations for examining and interpreting funnel plot] I have never seen one like, except when someone has constructed an artificial funnel plot to make a point.

funnel plot

Notice two things about this funnel plot. Rather than a smooth, unbroken distribution, studies with effect sizes between -.45 and +.45 are entirely missing. Studies with smaller sample sizes have the largest effect sizes, whereas the smallest effect sizes all come from the larger samples.

For me, this adds to the overwhelming evidence there is something gone wrong in this literature and any effect sizes should be ignored. There must have been considerable suppression of null findings so large effects from smaller studies will not generalize. Yet, the authors find the differences between small and larger sample studies encouraging

This suggests, encouragingly, that despite potential publication bias or inflationary bias due to neuroimaging analysis methods, nonetheless studies with larger samples tend to converge on similar and more reasonable (medium) effect sizes. Although such a conclusion is tentative, the results to date (Fig. 6) suggest that a sample size of approximately n = 25 is sufficient to reliably produce effect sizes that accord with those reported in studies with much larger samples (up to n = 46).

I and others have long argued that studies of this small sample size in evaluating psychotherapy should be left as pilot feasibility studies and not used to generate effect sizes. I think the same logic applies to this literature.

Distinctive patterns of regional activation and deactivation

The first part of the results section is devoted to studies examining particular forms of meditation. Seeing the apparent consistency of results, one needs to keep in mind the small number of studies being examined and the considerable differences among them. For instance, results presented for focused attention combine three between-group comparisons with four within-group studies. Focused attention includes both pre-post meditation differences from experienced Tibetan Buddhist practitioners to differences between novice and experienced practitioners of mindfulness-based stress reduction (MBSR). In almost all cases, meaningful statistically significant differences are found in both activation and deactivation regions that would make a lot of sense in terms of the functions that are known to be associated with them. There is not much noting of anomalous brain regions being identified by significant effects There is a high ratio of significant findings to number of participants comparisons. There is little discussion of anomalies.

Meta-analysis of focused attention studies resulted in 2 significant clusters of activation, both in prefrontal cortex (Table 3;Fig. 2). Activations were observed in regions associated with the voluntary regulation of thought and action, including the premotor cortex (BA 6; Fig. 2b) and dorsal anterior cingulate cortex (BA24; Fig. 2a). Slightly sub-threshold clusters were also observed in the dorsolateral prefrontal cortex (BA 8/9; Fig. 2c) and left midinsula (BA 13; Fig. 2e); we display these somewhat sub-threshod results here because of the obvious interest of these findings in practices that involve top-down focusing of attention, typically focused on respiration. We also observed clusters of deactivation in regions associated with episodic memory and conceptual processing, including the ventral posterior cingulate cortex (BA 31; Fig. 2d)and left inferior parietal lobule (BA 39; Fig. 2f).

How can such meaningful, practically significant findings obtains when so many conditions mitigate against finding them? John Ioannidis once remarked that in hot areas of research, consistency of positive findings from small studies often reflects only the strength of bias with which they are sought. The strength of findings will decrease when larger, more methodologically sophisticated studies become available, conducted by investigators who are less committed to having to get confirmation.

The article concludes:

Many have understandably viewed the nascent neuroscience of meditation with skepticism (Andresen, 2000; Horgan, 2004), burecent years have seen an increasing number of high-quality, controlled studies that are suitable for inclusion in meta-analyses and that can advance our cumulative knowledge of the neural basis of various meditation practices (Tang et al., 2015). With nearly a hundred functional neuroimaging studies of meditation now reported, we can conclude with some confidence that different practices show relatively distinct patterns of brain activity, and that the magnitude of associated effects on brain function may have some practical significance. The only totally incontrovertible conclusion, however, is that much work remains to be done to confirm and build upon these initial findings.

“Increasing number of high-quality, controlled studies that are suitable for inclusion in meta-analyses” ?…” “Conclude with some confidence…? “Relatively distinct patterns”?… “Some practical significance”?

In all of this premature enthusiasm about findings relating the practice of meditation to activation of particular regions of the brain and deactivation of others, we should not lose track of some other issues.

Although the authors talk about mapping one-to-one relationships between psychological states and regions of the brain, none of the studies would be of sufficient size to document some relationships, given the expected size of the relationship, based on what is typically found between psychological states and other biological variables.

Many differences between techniques could be artifactual –due to the technique altering breathing, involving verbalization, or focused attention. Observed differences in the brain regions activated and deactivated might simply reflect these differences without them being related to psychological functioning.

Even if the association were found, it would be a long way to establishing that the association reflected a causal mechanism, rather than simply being correlational or even artifactual. Think of the analogy of discovering a relationship between the amount of sweat while exercising in concluding that any weight loss was due to sweating it out.

We still have not established that meditation has more psychological and physical health benefits than other active interventions with presumably different mechanisms. After lots of studies, we still don’t know whether mindfulness meditation is anything more than a placebo. While I was finishing up this blog post, I came across a new study:

The limited prosocial effects of meditation: A systematic review and meta-analysis. 

Although we found a moderate increase in prosociality following meditation, further analysis indicated that this effect was qualified by two factors: type of prosociality and methodological quality. Meditation interventions had an effect on compassion and empathy, but not on aggression, connectedness or prejudice. We further found that compassion levels only increased under two conditions: when the teacher in the meditation intervention was a co-author in the published study; and when the study employed a passive (waiting list) control group but not an active one. Contrary to popular beliefs that meditation will lead to prosocial changes, the results of this meta-analysis showed that the effects of meditation on prosociality were qualified by the type of prosociality and methodological quality of the study. We conclude by highlighting a number of biases and theoretical problems that need addressing to improve quality of research in this area. [Emphasis added].

 

 

 

School-Based Mindfulness Based Stress-Reduction Program (MBSR) fails to deliver positive results

No positive effects found for Jon Kabat-Zinn’s Mindfulness Based Stress-Reduction Program with middle and high school students. Evidence of deterioration was found in some subgroup analyses.

mind the brain logo

No positive effects found for Jon Kabat-Zinn’s Mindfulness Based Stress-Reduction Program with middle and high School Students. Evidence of deterioration was found in some subgroup analyses.

mindfulness in schoolsWe should be cautious about interpreting negative effects that are confined to subgroup analyses. They may well be due to chance. But we should be concerned about the lack of positive findings across measures in the primary analyses. MBSR (a mindfulness training product trademarked and controlled by Jon Kabat-Zinn) and other mindfulness programs have heavily promoted as having wondrous benefits and mandated in many school settings.

The study [with link to the PDF]

Johnson C, Burke C, Brinkman S, Wade T. Effectiveness of a school-based mindfulness program for transdiagnostic prevention in young adolescents. Behaviour Research and Therapy. 2016 Jun 30;81:1-1.

Abstract

Anxiety, depression and eating disorders show peak emergence during adolescence and share common risk factors. School-based prevention programs provide a unique opportunity to access a broad spectrum of the population during a key developmental window, but to date, no program targets all three conditions concurrently. Mindfulness has shown promising early results across each of these psychopathologies in a small number of controlled trials in schools, and therefore this study investigated its use in a randomised controlled design targeting anxiety, depression and eating disorder risk factors together for the first time. Students (M age 13.63; SD = .43) from a broad band of socioeconomic demographics received the eight lesson, once weekly.b (“Dot be”) mindfulness in schools curriculum (N = 132) or normal lessons (N = 176). Anxiety, depression, weight/shape concerns and wellbeing were the primary outcome factors. Although acceptability measures were high, no significant improvements were found on any outcome at post-intervention or 3-month follow-up. Adjusted mean differences between groups at post-intervention were .03 (95% CI: -.06 to -.11) for depression, .01 (-.07 to -.09) for anxiety, .02 (-.05 to -.08) for weight/shape concerns, and .06 (-.08 to -.21) for wellbeing. Anxiety was higher in the mindfulness than the control group at follow-up for males, and those of both genders with low baseline levels of weight/shape concerns or depression. Factors that may be important to address for effective dissemination of mindfulness-based interventions in schools are discussed. Further research is required to identify active ingredients and optimal dose in mindfulness-based interventions in school settings.

The discussion noted:

The design of this study addresses several shortcomings identified in the literature (Britton et al., 2014; Burke, 2010; Felver et al., 2015; Meiklejohn et al., 2012; Tan, 2015; Waters et al., 2014). First, it was a multi-site, randomised controlled design with a moderately large sample size based on a priori power calculations. Second, it included follow-up (three months). Third, it sought to replicate an existing mindfulness-based intervention for youth. Fourth, socioeconomic status was not only reported but a broad range of socioeconomic bands included, although it was unfortunate that poor opt-in consent rates resulted in high data wastage in the lower range schools. Use of the same instructor for all classes in the intervention arm represents a strength (consistency) and a limitation (generalisability of findings).

Coverage in Scientific American

Mindfulness Training for Teens Fails Important Test

A large trial in schools showed no evidence of benefits, and hints it could even cause problems

The fact that this carefully-controlled investigation showed no benefits of mindfulness for any measure, and furthermore indicated an adverse effect for some participants, indicates that mindfulness training is not a universal solution for addressing anxiety or depression in teens, nor does it qualify as a replacement for more traditional psychotherapy or psychopharmacology, at least not as implemented in this school-based paradigm.

eBook_Mindfulness_345x550Preorders are being accepted for e-books providing skeptical looks at mindfulness and positive psychology, and arming citizen scientists with critical thinking skills. Right now there is a special offer for free access to a Mindfulness Master Class. But hurry, it won’t last.

I will also be offering scientific writing courses on the web as I have been doing face-to-face for almost a decade. I want to give researchers the tools to get into the journals where their work will get the attention it deserves.

Sign up at my website to get advance notice of the forthcoming e-books and web courses, as well as upcoming blog posts at this and other blog sites. Get advance notice of forthcoming e-books and web courses. Lots to see at CoyneoftheRealm.com.
 

“It’s certainly not bareknuckle:” Comments to a journalist about a critique of mindfulness research

We can’t assume authors of mindfulness studies are striving to do the best possible science, including being prepared for the possibility of being proven incorrect by their results.

mind the brain logo

I recently had a Skype interview with science journalist Peter Hess concerning an article in Psychological Science.

Peter was exceptionally prepared, had a definite point of view, but was open to what I said. In the end seem to be persuaded by me on a number of points.  The resulting article in Inverse  faithfully conveyed my perspective and juxtaposed quotes from me with those from an author of the Psych Science piece in a kind of debate.

My point of view

larger dogWhen evaluating an article about mindfulness in a peer-reviewed journal, we need to take into account that authors may not necessarily be striving to do the best science, but to maximally benefit their particular brand of mindfulness, their products, or the settings in which they operate. Many studies of mindfulness are a little more than infomercials, weak research intended only to get mindfulness promoters’ advertisement of themselves into print or to allow the labeling of claims as “peer-reviewed”. Caveat Lector.

We cannot assume authors of mindfulness studies are striving to do the best possible science, including being prepared for the possibility of being proven incorrect by their results. Rather they may be simply try to get the strongest possible claims through peer review, ignoring best research practices and best publication practices.

Psychologists Express Growing Concern With Mindfulness Meditation

“It’s not bare-knuckle, that’s for sure.”

There was much from the author of the Psych Science article with which  I would agree:

“In my opinion, there are far too many organizations, companies, and therapists moving forward with the implementation of ‘mindfulness-based’ treatments, apps, et cetera before the research can actually tell us whether it actually works, and what the risk-reward ratio is,” corresponding author and University of Melbourne research fellow Nicholas Van Dam, Ph.D. tells Inverse.

Bravo! And

“People are spending a lot of money and time learning to meditate, listening to guest speakers about corporate integration of mindfulness, and watching TED talks about how mindfulness is going to supercharge their brain and help them live longer. Best case scenario, some of the advertising is true. Worst case scenario: very little to none of the advertising is true and people may actually get hurt (e.g., experience serious adverse effects).”

But there were some statements that renewed the discomfort and disappointment I experienced when I read the original article in Psychological Science:

 “I think the biggest concern among my co-authors and I is that people will give up on mindfulness and/or meditation because they try it and it doesn’t work as promised,” says Van Dam.

“There may really be something to mindfulness, but it will be hard for us to find out if everyone gives up before we’ve even started to explore its best potential uses.”

So, how long before we “give up” on thousands of studies pouring out of an industry? In the meantime, should consumers act on what seem to be extravagant claims?

The Inverse article segued into some quotes from me after delivering another statement from the author which I could agree:

The authors of the study make their attitudes clear when it comes to the current state of the mindfulness industry: “Misinformation and poor methodology associated with past studies of mindfulness may lead public consumers to be harmed, misled, and disappointed,” they write. And while this comes off as unequivocal, some think they don’t go far enough in calling out specific instances of quackery.

“It’s not bare-knuckle, that’s for sure. I’m sure it got watered down in the review process,” James Coyne, Ph.D., an outspoken psychologist who’s extensively criticized the mindfulness industry, tells Inverse.

Coyne agrees with the conceptual issues outlined in the paper, specifically the fact that many mindfulness therapies are based on science that doesn’t really prove their efficacy, as well as the fact that researchers with copyrights on mindfulness therapies have financial conflicts of interest that could influence their research. But he thinks the authors are too concerned with tone policing.

“I do appreciate that they acknowledged other views, but they kept out anybody who would have challenged their perspective,” he says.

Regarding Coyne’s criticism about calling out individuals, Van Dam says the authors avoided doing that so as not to alienate people and stifle dialogue.

“I honestly don’t think that my providing a list of ‘quacks’ would stop people from listening to them,” says Van Dam. “Moreover, I suspect my doing so would damage the possibility of having a real conversation with them and the people that have been charmed by them.” If you need any evidence of this, look at David “Avocado” Wolfe, whose notoriety as a quack seems to make him even more popular as a victim of “the establishment.” So yes, this paper may not go so far as some would like, but it is a first step toward drawing attention to the often flawed science underlying mindfulness therapies.

To whom is the dialogue directed about unwarranted claims from the mindfulness industry?

As one of the authors of an article claiming to be an authoritative review from a group of psychologists with diverse expertise, Van Dam says he is speaking to consumers. Why won’t he and his co-authors provide citations and name names so that readers can evaluate for themselves what they are being told? Is the risk of reputational damage and embarrassment to the psychologists so great as to cause Van Dam to protect them versus protecting consumers from the exaggerated and even fraudulent claims of psychologists hawking their products branded as ‘peer-reviewed psychological and brain science’.

I use the term ‘quack’ sparingly outside of discussing unproven and unlikely-to-be-proven products supposed to promote physical health and well-being or to prevent or cure disease and distress.

I think Harvard psychologist Ellen Langer deserves the term “quack” for her selling of expensive trips to spas in Mexico to women with advanced cancer so that they can change their mind set to reverse the course of their disease. Strong evidence, please! Given that this self-proclaimed mother of mindfulness gets her claims promoted through the Association for Psychological Science website, I think it particularly appropriate for Van Dam and his coauthors to name her in their publication in an APS journal. Were they censored or only censoring themselves?

Let’s put aside psychologists who can be readily named as quacks. How about Van Dam and co-authors naming names of psychologists claiming to alter the brains and immune systems of cancer patients with mindfulness practices so that they improve their physical health and fight cancer, not just cope better with a life-altering disease?

I simply don’t buy Van Dam’s suggestion that to name names promotes quackery any more than I believe exposing anti-vaxxers promotes the anti-vaccine cause.

Is Van Dam only engaged in a polite discussion with fellow psychologists that needs to be strictly tone-policed to avoid offense or is he trying to reach, educate, and protect consumers as citizen scientists looking after their health and well-being? Maybe that is where we parted ways.

Will lessons in happiness solve the crisis in child mental health care?

bread and circussRome gave citizens bread and circuses. Is London giving citizens worthless randomized trials of inert interventions to solve the crisis of child mental health care without spending substantially more funds?

The UK  Department for Education (DfE) issued an Expression of Interest [ What’s that? ] for a large randomized trial comparing three preventive mental health interventions for promoting well-being among primary school children.

The three trialed interventions are:

Mindfulness

Mindfulness is the ability to direct attention to experience as it unfolds. It enables those who have learned it to be more able to be with their present experience, and respond more skilfully to whatever is happening. There is some evidence that it may be helpful in reducing anxiety, depressive symptoms and stress and improving wellbeing, attention, focus and cognitive skills.7 We know that mindfulness techniques are currently used by schools, with a range of existing programmes and approaches, but there is limited understanding of whether less intensive approaches work effectively in a school setting. The successful bidder will develop and trial a light touch (10-15 minute) intervention, comprising of simple exercises repeated at regular intervals (e.g. weekly or more than once a week) which provides teachers with materials to guide mindfulness practice e.g. audio tracks or guided exercises.

safety-net-PB-feeling-good-feeling-safe-resource-pack-a42Protective behaviours

Protective behaviours is a practical approach to personal safety, teaching children and young people to recognise early warning signs of not feeling safe and how to recognise where they can get help. It seeks to provide life skills, develop support structures and instil positive help seeking behaviours which can help keep children safe from a range of risks that may impact wellbeing and increase the risk of mental health problems. It is a well-established approach, with indications of ongoing use in schools, however evidence of effectiveness is limited. Some evidence suggests that it is beneficial for those at risk of mental health difficulties as well as the wider population, and it is relatively easy to integrate into the school environment. The successful bidder will develop and trial a light touch protective behaviours intervention which can easily be included in the school day, can be delivered by teachers/school staff to a whole class, with a small amount of training, and which builds on existing programmes and materials.

hands Relaxation and breathing-based techniques

Relaxation and breathing-based techniques and training for schools originated as targeted interventions to assist pupils with anxiety. However, there is emerging use of these approaches universally in primary schools, particularly in the form of short breathing exercises, with some reported increases in concentration, resilience, self-perception positivity and connection with others. There is currently limited evidence of wider use in schools or effectiveness, but there is a theoretical unpinning linking relaxation with improved wellbeing and engagement with learning. The successful bidder will develop and trial a light touch intervention that offers short regular exercises, delivered by teachers in the classroom with minimal training and materials, and which build on existing relation and breathing-based techniques.

Note that the requirement is that all three interventions be delivered in low intensity “light touch” versions, i.e, “easily be included in the school day, can be delivered by teachers/school staff to a whole class, with a small amount of training, and which builds on existing programmes and materials.”

The planned trial is ambitious and large-scale, involving:

  • Recruitment of 100 volunteer primary schools…representing a range of different school types, locations and demographics.
  • Even randomization of schools into one of three arms corresponding to the three interventions, with 33 schools in each arm.
  • Classes in each school evenly randomized to intervention or control group.
  • A small amount of funding would help cover costs of participation and to incentivise full engagement with the trial.:

Final selection of primary and secondary outcomes are left to applicants, but expected to include short measures of

  • Subjective Wellbeing
  • Mental health/psychological wellbeing
  • Engagement with education

The larger context

The expression of interest was a follow-up to “The Shared Society”, UK Prime Minister Theresa May’s recent speech at the Charity Commission. In that speech the Prime Minister identified “the burning injustice of mental illness” and stated:

“This is an historic opportunity to right a wrong, and give people deserving of compassion and support the attention and treatment they deserve. And for all of us to change the way we view mental illness so that striving to improve mental wellbeing is seen as just as natural, positive and good as striving to improve our physical wellbeing.”

However, the Independent noted:

The speech however barely announces any extra cash to improve underfunded services – with just an extra £15m expected to be pledged for creating “places of safety”. This amounts to about £23,000 per parliamentary constituency.

Research conducted by the Education Policy Institute Independent Commission on Children and Young People’s Mental Health in November found that a quarter of young people seeking mental health care are turned away by specialist services because of a lack of resources. Waiting times for treatment in many areas are also incredibly long.

The House of Commons Public Accounts Committee said in September that it was “sceptical” about the Government’s attempt to improve mental health services without a significant amount of extra cash.

Praise for the speech

Nonetheless, the Independent reported praise to the Prime Minister’s speech:

Paul Farmer, chief executive of Mind, the mental health charity, said it was good that the Prime Minister was talking about mental health.

“It’s important to see the Prime Minister talking about mental health and shows how far we have come in bringing the experiences of people with mental health problems up the political agenda,” he said.

“Mental health should be at the heart of government, and at the heart of society and communities – it’s been on the periphery for far too long.”

He said he welcomed the focus on prevention in schools and workplaces and support for people in crisis.

Sir Ian Cheshire, chairman of the Heads Together Campaign described the Prime Minister’s announcements as “extremely important and very welcome”.

“They show both a willingness to tackle the broad challenge of mental health support and a practical grasp of how to start making a real difference,” he said.

As I noted in another blog post, the Heads Together Campaign is an initiative of the Royals.

Paul Farmer, chief executive of Mind, the mental health charity, said it was good that the Prime Minister was talking about mental health.

“It’s important to see the Prime Minister talking about mental health and shows how far we have come in bringing the experiences of people with mental health problems up the political agenda,” he said.

“Mental health should be at the heart of government, and at the heart of society and communities – it’s been on the periphery for far too long.”

He said he welcomed the focus on prevention in schools and workplaces and support for people in crisis.

Sir Ian Cheshire, chairman of the Heads Together Campaign described the Prime Minister’s announcements as “extremely important and very welcome”.

“They show both a willingness to tackle the broad challenge of mental health support and a practical grasp of how to start making a real difference,” he said.

Praise for the interventions that were selected for evaluation

 An article in The Guardian reported praise for the interventions that were selected for evaluation:

Laura Henry, an early years consultant and Ofsted inspector, said the trials could save the government billions in social care and housing costs down the line. “I think it’s an excellent idea,” she said. “Over the last decade there has been a massive push to academia, results and school league tables and children’s personal social development has been left behind.

“A holistic approach is needed and children should be able to self-regulate their own behaviour.”

Henry, a former teacher whose elder son is on the autistic spectrum, said specially trained teachers should help with grieving techniques and that any questions about bullying and pupils’ friends needed to be sensitive.

“It’s absolutely the best way to spend DfE money,” she said. “It will save x amount of money in social care when they are adults.”

And:

The mindfulness trial was welcomed by the educational pioneer Sir Anthony Seldon, who was pooh-poohed when he brought in such classes while master of the private school Wellington College. He said: “It was negligent of government [in the past] to have this unintelligent response to wellbeing, saying this was la-la land and psychobabble. We have a crisis in mental health which is reducible now that government is beginning to take seriously the right interventions to look after the wellbeing of young people.”

Professor Alan Smithers, of the University of Buckingham, where Seldon is vice-chancellor, was more sceptical. He said: “It is good the government is having a trial and not rushing in. There are so many demands on resources for schools that it is important we know that mindfulness lessons work. “There are many calls on school funding: the need for teachers, the squeeze on budgets and school buildings.”

Lord Layard, Britain’s “happiness guru”, and Lord O’Donnell, the former cabinet secretary, will meet the government this month to discuss how to enable schools to measure children’s wellbeing as a guide to performance. They want schools to give similar weight to children’s happiness as to their academic results. Under their proposals, schools would be measured on whether pupils’ happiness improved or declined. Children would fill in a questionnaire asking, among other things, whether “I have at least one good friend”; “other people generally like me”; “other people pick on me or bully me”; and “I would rather be alone than with others”.

Pupils’ scores would be confidential, but could be used to alert the school to serious difficulties. NHS workers would provide psychological treatment to children in schools at short notice before they became so ill that they qualified for admission to mental health services.

The interventions are unlikely to improve mental health comes, even self-reported well-being, and may prove harmful.

I dare you to be positive Tssk! The UK has some talented mental health services researchers. Why aren’t we hearing their collective voices of outrage about a useless trial squandering millions of pounds, potentially harming schools and students, and mainly serving to distract from the government’s lack of action to correct the underfunding of both mental health care for children and the school systems?

Instead, we have some self-proclaimed authorities waxing enthusiastically. As a group, they are lacking in mental health training and serve to benefit immensely from these initiatives. Journalists should get them out of the picture or at least better reveal the conflicts of interest and balance their commentary with comments that are more evidence driven.

Even when delivered with full intensity, the interventions lack evidence of effectiveness needed to justify a large-scale trial. Yet The Expression of Interest specifies that they be delivered in a lite form – only a few minutes a week . This is Unlikely to improve the measured outcomes or impact on effective use of already scarce child mental health services with an acceptably long wait times.

Funding the trial is a poor substitute for better funding for mental health services and schools. Yet politicians and policymakers can point to them and argue that the UK is conducting the research desperately needed to address these issues and so we need to be patient.

I’m not sure we should consider these trials as serious attempts to contribute to the mental health services literature. Selection of the particular interventions to be trialed seems to be political and tied to what is already being done in some schools. Their existing implementation likely reflects vested interests that undoubtedly influenced the selection for trialing and hope to benefit financially from the opportunities it will provide. I don’t think that the mere fact that interventions are already in use justifies an ambitious and expensive evaluation of them unless there is further evidence that they are likely to be effective.

The Expression of Interest cites one review of mindfulness studies. I looked it up and it is unusually candid in indicating the limitations in quality and quantity of relevant studies that mindfulness training can affect such outcomes. The review stands in sharp contrast to the unbalanced and prematurely enthusiastic Mindful Nation UK report.

We should have serious concerns about the lack of evidence that Protective Behaviours could have any effect on the outcomes selected to evaluate the programs. Conceivably, it could do some harm to at risk children. Getting children to disclose bullying and frank abuse at home and school can only aggravate these problem and invite retaliation if effective intervention is not available to address these problems. I would be curious to know the extent to which primary school teachers are already aware of such problems but lack the tools or time to address them.

Basically, Protective Behaviours is a kind of screening program facilitated by encouragement to disclosed. Such programs can prove ineffective if they do not occur in a system prepared to quickly offer affective interventions. Such screening programs can compete for scarce resources that would otherwise be used to deal with already known problems requiring more intensive and focused intervention.

There is the precedent of GPs screening women for domestic abuse. Routine screening seemed to address a documented ignoring of the problem. However, the World Health Organization (WHO) withdrew the recommendation because of lack of any evidence that improved health outcomes for women and summoned consistent evidence that at least some women were harmed by ineffectual interventions that heighten the abuse that they were receiving.

The breathing and relaxation exercises might conceivably be a nonspecific control condition, except that all of the inventions are untried, lacking in evidence, and delivered in such a low intensity that they themselves are best nonspecific control conditions. I think it’s inconceivable that meaningful differences will be demonstrated among the three interventions. At best, the trials can conclude that they are equally effective or not effective at all. The question whether these interventions are better than other active interventions or other deployments of scarce resources left unaddressed.

eBook_PositivePsychology_345x550I will soon be offering e-books providing skeptical looks at mindfulness and positive psychology, as well as scientific writing courses on the web as I have been doing face-to-face for almost a decade.

Sign up at my new website to get advance notice of the forthcoming e-books and web courses, as well as upcoming blog posts at this and other blog sites.  Get advance notice of forthcoming e-books and web courses. Lots to see at CoyneoftheRealm.com.

Misleading systematic review of mindfulness studies used to promote Bensen Institute for Mind-Body Medicine services

A seriously flawed overview “systematic review “ of systematic reviews and meta-analyses of the effects of mindfulness on health and well-being alerts readers how they need to be skeptical of what they are told about the benefits of mindfulness.

Especially when the information comes those benefiting enormously from promoting the practice.

The glowing evaluation of the benefits of mindfulness presented in a PLOS One review is contradicted by a more comprehensive and systematic review which was cited but summarily dismissed. As we will see, the PLOS One article sidesteps substantial confirmation bias and untrustworthiness in the mindfulness literature.

The review was prepared by authors associated with the Benson-Henry Institute for Mind-Body Medicine, which is tied to Massachusetts General Hospital and Harvard Medical School. The institute directly markets mindfulness treatment to patients and training to professionals and organizations.  Its website provides links to research articles such as this one, which are used to market a wide range of programs –

being calm

Recently PLOS One published corrections to five articles from this group concerning previous statements about the authors having no conflicts of interest to declare. The corrections acknowledged extensive conflicts of interest.

The Competing Interests statement is incorrect. The correct Competing Interests statement is: The following authors hold or have held positions at the Benson-Henry Institute for Mind-Body Medicine at Massachusetts General Hospital, which is paid by patients and their insurers for running the SMART-3RP and related relaxation/mindfulness clinical programs, markets related products such as books, DVDs, CDs and the like, and holds a patent pending (PCT/US2012/049539 filed August 3, 2012) entitled “Quantitative Genomics of the Relaxation Response.”

While the review we will be discussing was not corrected, it should have been.

The same conflicts of interest should have been disclosed to readers evaluating the trustworthiness of what is being presented to them.

Probing this review will demonstrate just how hard it is to uncover the bias and distortions that routinely is provided by promoters of mindfulness wanting to demonstrate the evidence base for what they offer.

The article is

Gotink, R.A., Chu, P., Busschbach, J.J., Benson, H., Fricchione, G.L. and Hunink, M.M., 2015. Standardised mindfulness-based interventions in healthcare: an overview of systematic reviews and meta-analyses of RCTs. PLOS One, 10(4), p.e0124344.

The abstract offers the conclusion:

The evidence supports the use of MBSR and MBCT to alleviate symptoms, both mental and physical, in the adjunct treatment of cancer, cardiovascular disease, chronic pain, depression, anxiety disorders and in prevention in healthy adults and children.

This evaluation is more emphatically stated near the end of the article:

This review provides an overview of more trials than ever before and the intervention effect has thus been evaluated across a broad spectrum of target conditions, most of which are common chronic conditions. Study settings in many countries across the globe contributed to the analysis, further serving to increase the generalizability of the evidence. Beneficial effects were mostly seen in mental health outcomes: depression, anxiety, stress and quality of life improved significantly after training in MBSR or MBCT. These effects were seen both in patients with medical conditions and those with psychological disorders, compared with many types of control interventions (WL, TAU or AT). Further evidence for effectiveness was provided by the observed dose-response relationship: an increase in total minutes of practice and class attendance led to a larger reduction of stress and mood complaints in four reviews [18,20,37,54].

Are you impressed? “More than ever before”? “Generalizability of the evidence”? Really?

And in wrap up summary comments:

Although there is continued scepticism in the medical world towards MBSR and MBCT, the evidence indicates that MBSR and MBCT are associated with improvements in depressive symptoms, anxiety, stress, quality of life, and selected physical outcomes in the adjunct treatment of cancer, cardiovascular disease, chronic pain, chronic somatic diseases, depression, anxiety disorders, other mental disorders and in prevention in healthy adults and children.

Compare and contrast these conclusions with a more balanced and comprehensive review.

The US Agency for Healthcare Research and Quality (AHCRQ) commissioned a report from Johns Hopkins University Evidence-based Practice Center.

The 439 page report is publicly available:

Goyal M, Singh S, Sibinga EMS, Gould NF, Rowland-Seymour A, Sharma R, Berger Z, Sleicher D, Maron DD, Shihab HM, Ranasinghe PD, Linn S, Saha S, Bass EB, Haythornthwaite JA. Meditation Programs for Psychological Stress and Well-Being. Comparative Effectiveness Review No. 124. (Prepared by Johns Hopkins University Evidence-based Practice Center under Contract No. 290-2007-10061–I.) AHRQ Publication No. 13(14)-EHC116-EF. Rockville, MD: Agency for Healthcare Research and Quality; January 2014.

A companion, less detailed article was also published in JAMA: Internal Medicine:

Goyal, M., Singh, S., Sibinga, E.M., Gould, N.F., Rowland-Seymour, A., Sharma, R., Berger, Z., Sleicher, D., Maron, D.D., Shihab, H.M. and Ranasinghe, P.D., 2014. Meditation programs for psychological stress and well-being: a systematic review and meta-analysis. JAMA Internal Medicine, 174(3), pp.357-368.

Consider how conclusions of this article were characterized in the Bensen-Henry PLOS One article. The article is briefly mentioned without detailing its methods and conclusions.

Recently, Goyal et al. published a review of mindfulness interventions compared to active control and found significant improvements in depression and anxiety[7].

And

A recent review compared meditation to only active control groups, and although lower, also found a beneficial effect on depression, anxiety, stress and quality of life. This review was excluded in our study for its heterogeneity of interventions [7].

What the Goyal et JAMA: Internal Medicine actually said:

After reviewing 18 753 citations, we included 47 trials with 3515 participants. Mindfulness meditation programs had moderate evidence of improved anxiety (effect size, 0.38 [95% CI, 0.12-0.64] at 8 weeks and 0.22 [0.02-0.43] at 3-6 months), depression (0.30 [0.00-0.59] at 8 weeks and 0.23 [0.05-0.42] at 3-6 months), and pain (0.33 [0.03- 0.62]) and low evidence of improved stress/distress and mental health–related quality of life. We found low evidence of no effect or insufficient evidence of any effect of meditation programs on positive mood, attention, substance use, eating habits, sleep, and weight. We found no evidence that meditation programs were better than any active treatment (ie, drugs, exercise, and other behavioral therapies).

The review also notes that evidence of the effectiveness mindfulness interventions is largely limited to trials in which it is compared to no treatment, wait list, or a usually ill-defined treatment as usual (TAU).

In our comparative effectiveness analyses (Figure 1B), we found low evidence of no effect or insufficient evidence that any of the meditation programs were more effective than exercise, progressive muscle relaxation, cognitive-behavioral group therapy, or other specific comparators in changing any outcomes of interest. Few trials reported on potential harms of meditation programs. Of the 9 trials reporting this information, none reported any harms of the intervention.

This solid JAMA: Internal Medicine review explains why its conclusions may differ from past reviews:

Reviews to date report a small to moderate effect of mindfulness and mantra meditation techniques in reducing emotional symptoms (eg, anxiety, depression, and stress) and improving physical symptoms (eg, pain).7– 26 These reviews have largely included uncontrolled and controlled studies, and many of the controlled studies did not adequately control for placebo effects (eg, waiting list– or usual care–controlled studies). Observational studies have a high risk of bias owing to problems such as self-selection of interventions (people who believe in the benefits of meditation or who have prior experience with meditation are more likely to enroll in a meditation program and report that they benefited from one) and use of outcome measures that can be easily biased by participants’ beliefs in the benefits of meditation. Clinicians need to know whether meditation training has beneficial effects beyond self-selection biases and the nonspecific effects of time, attention, and expectations for improvement.27,28

Basically, this article insists that mindfulness be evaluated in a  head-to- head comparison to an active treatment. Failure to provide such a comparison means not being able to rule out that apparent effects of mindfulness are nonspecific, i.e.,  not due to any active ingredient of the practice.

An accompanying editorial commentary raised troubling issues about the state of the mindfulness literature. It noted that limiting inclusion to RCTs with an active control condition and a patient population experiencing mental or physical health problems left only 3% (47/18,753 of the citations that had been retrieved. Furthermore:

The modest benefit found in the study by Goyal et al begs the question of why, in the absence of strong scientifically vetted evidence, meditation in particular and complementary measures in general have become so popular, especially among the influential and well educated…What role is being played by commercial interests? Are they taking advantage of the public’s anxieties to promote use of complementary measures that lack a base of scientific evidence? Do we need to require scientific evidence of efficacy and safety for these measures?

How did the Bensen-Henry review arrive at a more favorable assessment?

The issue that dominated the solid Goyal et al systematic review and meta analysis is not prominent in the Bensen-Henry review. The latter article hardly mentions the importance of whether mindfulness is compared to an active treatment. It doesn’t mention if any difference in effect size for mindfulness can be expected when the comparison is an active treatment.

The Bensen-Henry review stated that it excluded systematic reviews and meta analyses if they did not focus on MBCT or MBSR. One has to search the supplementary materials to find that Goyal et al was excluded because it did not calculate separate effect sizes for mindfulness-based stress reduction (MBSR).

However, Bensen-Henry review included narrative systematic reviews that did not calculate effect sizes at all. Furthermore, the excluded Goyal et al JAMA: Internal Medicine article summarized MBSR separate from other forms of meditation and the more comprehensive AHCQR report provided detailed forest plots of effect sizes for MBSR with specific outcomes and patient populations.

Hmm, keeping out evidence that does fit with the sell-job story?

We need to keep in mind the poor manner in which MBSR was specified, particularly in the early studies that dominate the reviews covered by the Bensen – Henry article. Many of the treatments were not standardized and certainly not manualized. They sometimes, but not always incorporate psychoeducation, other cognitive behavioral techniques, and varying types of yoga.

The Bensen-Henry authors claimed to have performed quality assessments  of the reviews  included using a checklist based on the validated PRISMA guidelines. However, PRISMA evaluates the quality of reporting in reviews, not the quality of how the review was done. The checklist used by the Bensen-Henry authors was highly selective in terms of which PRISMA items it chose to include, left unvalidated, and simply eccentric. For instance, one item evaluated a review favorably if it interpreted studies “independent of funding source.”

A lack of independence of a study from its funding source is generally considered a high risk of bias.  There is ample documentation of  industry-funded studies and reviews exaggerating the efficacy of interventions supported by industry.

Our group received the Bill Silverman Prize from the Cochrane Collaboration for our identifying funding source as an overlooked source of bias in many meta analyses and, in particular, in Cochrane reviews. The Bensen-Henry checklist scores a review ignoring funding source as a virtue, not a vice! These authors are letting trials and reviews from promoters of mindfulness off the hook for potential conflict of interest, including their own studies and this review.

Examination of the final sample of reviews included in the Bensen-Henry analysis reveals that some are narrative reviews and could not contribute effect sizes. Some are older reviews that depend on a less developed literature. While optimistic about the promise of mindfulness, authors of these reviews frequently complained about the limits on the quantity and quality of available studies, calling for larger and better quality studies. When integrated and summarized by the Bensen-Henry authors, these reviews were given a more positive glow than the original authors conveyed.

Despite claims of being an “overview of more trials than ever before”, Bensen-Henry excluded all but 23 reviews. Some of those included do not appear to be recent or rigorous, particularly when contrasted with the quality and rigor of the excluded Goyal et al:

MJ, Norris RL, Bauer-Wu SM (2006) Mindfulness meditation for oncology patients: A discussion and critical review. Integr Cancer Ther 5: 98–108. pmid:16685074

Shennan C, Payne S, Fenlon D (2011) What is the evidence for the use of mindfulness-based interventions in cancer care? A review. Psycho-Oncology 20: 681–697.

Veehof MM, Oskam MJ, Schreurs KMG, Bohlmeijer ET (2011) Acceptance-based interventions for the treatment of chronic pain: A systematic review and meta-analysis. Pain 152: 533–542

Coelho HF, Canter PH, Ernst E (2007) Mindfulness-Based Cognitive Therapy: Evaluating Current Evidence and Informing Future Research. J Consult Clin Psychol 75: 1000–1005.

Ledesma D, Kumano H (2009) Mindfulness-based stress reduction and cancer: A meta-analysis. Psycho-Oncology 18: 571–579.

Ott MJ, Norris RL, Bauer-Wu SM (2006) Mindfulness meditation for oncology patients: A discussion and critical review. Integr Cancer Ther 5: 98–108.

Burke CA (2009) Mindfulness-Based Approaches with Children and Adolescents: A Preliminary Review of Current Research in an Emergent Field. J Child Fam Stud.

Do we get the most authoritative reviews of mindfulness from  Holist Nurs Pract, Integr Cancer Ther, and Psycho-Oncology?

To cite just one example of the weakness of evidence being presented as strong, take the bold Bensen-Henry conclusion:

Further evidence for effectiveness was provided by the observed dose-response relationship: an increase in total minutes of practice and class attendance led to a larger reduction of stress and mood complaints in four reviews [18,20,37,54].

“Observed dose-response relationship”? This claim is  based [check out with respect to the citations just above] on Ott et al, 18, Smith et al 20, Burke 37 and Proulx 54, which makes the evidence neither recent nor systematic. I am confident that other examples will not hold up if scrutinized.

Further contradiction of the too perfect picture of mindfulness therapy conveyed by the Bensen – Henry review.

A more recent PLOS One review of mindfulness studies exposed the confirmation bias in the published mindfulness literature. It suggested a too perfect picture has been created of uniformly positive studies.

Coronado-Montoya, S., Levis, A.W., Kwakkenbos, L., Steele, R.J., Turner, E.H. and Thombs, B.D., 2016. Reporting of positive results in randomized controlled trials of mindfulness-based mental health interventions. PLOS One, 11(4), p.e0153220.

A systematic search yielded 124 RCTs of mindfulness-based treatments:

108 (87%) of 124 published trials reported >1 positive outcome in the abstract, and 109(88%) concluded that mindfulness-based therapy was effective, 1.6 times greater than the expected number of positive trials based on effect size d = 0.55 (expected number positivetrials = 65.7). Of 21 trial registrations, 13 (62%) remained unpublished 30 months post-trial completion.

Furthermore:

None of the 21 registrations, however, adequately specified a single primary outcome (or multiple primary outcomes with an appropriate plan for statistical adjustment) and specified the outcome measure, the time of assessment, and the metric (e.g., continuous, dichotomous). When we removed the metric requirement, only 2 (10%) registrations were classified as adequate.

And finally:

There were only 3 trials that were presented unequivocally as negative trials without alternative interpretations or caveats to mitigate the negative results and suggest that the treatment might still be an effective treatment.

What we have is a picture of trials of mindfulness-based treatment having an excess of positive studies, given the study sample sizes. Selective reporting of positive outcomes likely contributed to this excess of published positive findings in the published literature. Most of the trials were not preregistered and so it’s unclear whether the positive outcomes that were reported were hypothesized to be the primary outcomes of interest. Most of the trials that were preregistered remained unpublished 30 months after the trials were completed.

The Goyal et al. study originally planned to conduct quantitative analyses of publication biases, but abandoned the effort when they couldn’t find sufficient numbers of the 47 studies that that reported most of the outcomes they evaluated.

Conclusion

 The Bensen-Henry review produces a glowing picture of the quality of RCTs evaluating MSBR and the consistency of positive findings across diverse outcomes and populations. This is consistent with the message that they want to promote in marketing their products to patients, clinicians, and institutions. In this blog post I’ve uncovered substantial problems in internal to the Bensen-Henry review in terms of the studies that were included and the manner in which they were evaluated. But now we have external evidence in two reviews without obvious conflicts of interest come into markedly different appraisals of a literature that lacks appropriate control groups and seems to be reporting findings with a distinct confirmation bias.

I could have gone further, but what I found about the Bensen-Henry review seems sufficient for a serious challenge to the validity of its conclusions.  Investigation of the claims made about dose-response relationships between amount of mindfulness practice and outcomes should encourage probing of other specific claims.

The larger issue is that we should not rely on promoters of MSBR products to provide unbiased estimates of their efficacy. This issue recalls very similar problems in the evaluation of Triple P Parenting Programs. Evaluations in which promoters were involved produce markedly more positive results than from independent evaluations. Exposure by my colleagues and me led to over 50 corrections and corrigendum to articles that previously had no conflicts of interest. But the process did not occur without fierce resistance from those whose livelihood was being challenged.

A correction to the Bensen-Henry PLOS One review is in order to clarify the obvious conflicts of interest of the authors. But the problem is not limited to reviews or original studies from Benson-Henry Institute for Mind-Body Medicine. It’s time that authors be required to answer more explicit questions about conflict of interest. Ruling out a conflict of interest should be based on authors having to endorse explicitly no conflicts, rather than on their basis of their not disclosing a conflict and then being able to claim it was an oversight that they did not report one.

Postscript Who was watching at PLOS One to keep out infomercials from promoters associated with Massachusetts General Hospital and Harvard Medical School? The Academic Editor was To avoid the appearance of  a conflict of interest,  should he have recused him from serving as editor?

This is another flawed paper for which I’d love to see the reviews.

eBook_Mindfulness_345x550I will soon be offering e-books providing skeptical looks at mindfulness and positive psychology, as well as scientific writing courses on the web as I have been doing face-to-face for almost a decade.

Sign up at my new website to get advance notice of the forthcoming e-books and web courses, as well as upcoming blog posts at this and other blog sites.  Get advance notice of forthcoming e-books and web courses. Lots to see at CoyneoftheRealm.com.

 

Unintended consequences of universal mindfulness training for schoolchildren?

the mindful nationThis is the first installment of what will be a series of occasional posts about the UK Mindfulness All Party Parliamentary Group report,  Mindful Nation.

  • Mindful Nation is seriously deficient as a document supposedly arguing for policy based on evidence.
  • The professional and financial interests of lots of people involved in preparation of the document will benefit from implementation of its recommendations.
  • After an introduction, I focus on two studies singled in Mindful Nation out as offering support for the benefits of mindfulness training for school children.
  • Results of the group’s cherrypicked studies do not support implementation of mindfulness training in the schools, but inadvertently highlight some issues.
  • Investment in universal mindfulness training in the schools is unlikely to yield measurable, socially significant results, but will serve to divert resources from schoolchildren more urgently in need of effective intervention and support.
  • Mindfulness Nation is another example of  delivery of  low intensity  services to mostly low risk persons to the detriment of those in greatest and most urgent need.

The launch event for the Mindful Nation report billed it as the “World’s first official report” on mindfulness.

Mindful Nation is a report written by the UK Mindfulness All-Party Parliamentary Group.

The Mindfulness All-Party Parliamentary Group (MAPPG)  was set up to:

  • review the scientific evidence and current best practice in mindfulness training
  • develop policy recommendations for government, based on these findings
  • provide a forum for discussion in Parliament for the role of mindfulness and its implementation in public policy.

The Mindfulness All-Party Parliamentary Group describes itself as

Impressed by the levels of both popular and scientific interest, and launched an inquiry to consider the potential relevance of mindfulness to a range of urgent policy challenges facing government.

Don’t get confused by this being a government-commissioned report. The report stands in sharp contrast to one commissioned by the US government in terms of unbalanced constitution of the committee undertaking the review, and lack  of transparency in search for relevant literature,  and methodology for rating and interpreting of the quality of available evidence.

ahrq reportCompare the claims of Mindful Nation to a comprehensive systematic review and meta-analysis prepared for the US Agency for Healthcare Research and Quality (AHRQ) that reviewed 18,753 citations, and found only 47 trials (3%) that included an active control treatment. The vast majority of studies available for inclusion had only a wait list or no-treatment control group and so exaggerated any estimate of the efficacy of mindfulness.

Although the US report was available to those  preparing the UK Mindful Nation report, no mention is made of either the full contents of report or a resulting publication in a peer-reviewed journal. Instead, the UK Mindful Nation report emphasized narrative and otherwise unsystematic reviews, and meta-analyses not adequately controlling for bias.

When the abridged version of the AHRQ report was published in JAMA: Internal Medicine, an accompanying commentary raises issues even more applicable to the Mindful Nation report:

The modest benefit found in the study by Goyal et al begs the question of why, in the absence of strong scientifically vetted evidence, meditation in particular and complementary measures in general have become so popular, especially among the influential and well educated…What role is being played by commercial interests? Are they taking advantage of the public’s anxieties to promote use of complementary measures that lack a base of scientific evidence? Do we need to require scientific evidence of efficacy and safety for these measures?

The members of the UK Mindfulness All-Party Parliamentary Group were selected for their positive attitude towards mindfulness. The collection of witnesses they called to hearings were saturated with advocates of mindfulness and those having professional and financial interests in arriving at a positive view. There is no transparency in terms of how studies or testimonials were selected, but the bias is notable. Many of the scientific studies were methodologically poor, if there was any methodology at all. Many were strongly stated, but weakly substantiated opinion pieces. Authors often included those having  financial interests in obtaining positive results, but with no acknowledgment of conflict of interest. The glowing testimonials were accompanied by smiling photos and were unanimous in their praise of the transformative benefits of mindfulness.

As Mark B. Cope and David B. Allison concluded about obesity research, such a packing of the committee and a highly selective review of the literature leads to a ”distortion of information in the service of what might be perceived to be righteous ends.” [I thank Tim Caulfield for calling this quote to my attention].

Mindfulness in the schools

The recommendations of Mindfulness Nation are

  1. The Department for Education (DfE) should designate, as a first step, three teaching schools116 to pioneer mindfulness teaching,co-ordinate and develop innovation, test models of replicability and scalability and disseminate best practice.
  2. Given the DfE’s interest in character and resilience (as demonstrated through the Character Education Grant programme and its Character Awards), we propose a comparable Challenge Fund of £1 million a year to which schools can bid for the costs of training teachers in mindfulness.
  3. The DfE and the Department of Health (DOH) should recommend that each school identifies a lead in schools and in local services to co-ordinate responses to wellbeing and mental health issues for children and young people117. Any joint training for these professional leads should include a basic training in mindfulness interventions.
  4. The DfE should work with voluntary organisations and private providers to fund a freely accessible, online programme aimed at supporting young people and those who work with them in developing basic mindfulness skills118.
Payoff of Mindful Nation to Oxford Mindfulness Centre will be huge.
Payoff of Mindful Nation to Oxford Mindfulness Centre will be huge.

Leading up to these recommendations, the report outlined an “alarming crisis” in the mental health of children and adolescents and proposes:

Given the scale of this mental health crisis, there is real urgency to innovate new approaches where there is good preliminary evidence. Mindfulness fits this criterion and we believe there is enough evidence of its potential benefits to warrant a significant scaling-up of its availability in schools.

Think of all the financial and professional opportunities that proponents of mindfulness involved in preparation of this report have garnered for themselves.

Mindfulness to promote executive functioning in children and adolescents

For the remainder of the blog post, I will focus on the two studies cited in support of the following statement:

What is of particular interest is that those with the lowest levels of executive control73 and emotional stability74 are likely to benefit most from mindfulness training.

The terms “executive control” and “emotional stability” were clarified:

Many argue that the most important prerequisites for child development are executive control (the management of cognitive processes such as memory, problem solving, reasoning and planning) and emotion regulation (the ability to understand and manage the emotions, including and especially impulse control). These main contributors to self-regulation underpin emotional wellbeing, effective learning and academic attainment. They also predict income, health and criminality in adulthood69. American psychologist, Daniel Goleman, is a prominent exponent of the research70 showing that these capabilities are the biggest single determinant of life outcomes. They contribute to the ability to cope with stress, to concentrate, and to use metacognition (thinking about thinking: a crucial skill for learning). They also support the cognitive flexibility required for effective decision-making and creativity.

Actually, Daniel Goleman is the former editor of the pop magazine Psychology Today and an author of numerous pop books.

The first cited paper.

73 Flook L, Smalley SL, Kitil MJ, Galla BM, Kaiser-Greenland S, Locke J, et al. Effects of mindful  awareness practices on executive functions in elementary school children. Journal of Applied School Psychology. 2010;26(1):70-95.

Journal of Applied School Psychology is a Taylor-Francis journal, formerly known as Special Services in the Schools (1984 – 2002).  Its Journal Impact Factor is 1.30.

One of the authors of the article, Susan Kaiser-Greenland is a mindfulness entrepreneur as seen in her website describing her as an author, public speaker, and educator on the subject of sharing secular mindfulness and meditation with children and families. Her books are The Mindful Child: How to Help Your Kid Manage Stress and Become Happier, Kinder, and More Compassionate and Mindful Games: Sharing Mindfulness and Meditation with Children, Teens, and Families and the forthcoming The Mindful Games Deck: 50 Activities for Kids and Teens.

This article represents the main research available on Kaiser-Greenfield’s Inner Kids program and figures prominently in her promotion of her products.

The sample consisted of 64 children assigned to either mindful awareness practices (MAPs; n = 32) or a control group consisting of a silent reading period (n = 32).

The MAPs training used in the current study is a curriculum developed by one of the authors (SKG). The program is modeled after classical mindfulness training for adults and uses secular and age appropriate exercises and games to promote (a) awareness of self through sensory awareness (auditory, kinesthetic, tactile, gustatory, visual), attentional regulation, and awareness of thoughts and feelings; (b) awareness of others (e.g., awareness of one’s own body placement in relation to other people and awareness of other people’s thoughts and feelings); and (c) awareness of the environment (e.g., awareness of relationships and connections between people, places, and things).

A majority of exercises involve interactions among students and between students and the instructor.

Outcomes.

The primary EF outcomes were the Metacognition Index (MI), Behavioral Regulation Index (BRI), and Global Executive Composite (GEC) as reported by teachers and parents

Wikipedia presents the results of this study as:

The program was delivered for 30 minutes, twice per week, for 8 weeks. Teachers and parents completed questionnaires assessing children’s executive function immediately before and following the 8-week period. Multivariate analysis of covariance on teacher and parent reports of executive function (EF) indicated an interaction effect baseline EF score and group status on posttest EF. That is, children in the group that received mindful awareness training who were less well regulated showed greater improvement in EF compared with controls. Specifically, those children starting out with poor EF who went through the mindful awareness training showed gains in behavioral regulation, metacognition, and overall global executive control. These results indicate a stronger effect of mindful awareness training on children with executive function difficulties.

The finding that both teachers and parents reported changes suggests that improvements in children’s behavioral regulation generalized across settings. Future work is warranted using neurocognitive tasks of executive functions, behavioral observation, and multiple classroom samples to replicate and extend these preliminary findings.”

What I discovered when I scrutinized the study.

 This study is unblinded, with students and their teachers and parents providing the subjective ratings of the students well aware of which group students are assigned. We are not given any correlations among or between their ratings and so we don’t know whether there is just a global subjective factor (easy or difficult child, well-behaved or not) operating for either teachers or parents, or both.

It is unclear for what features of the mindfulness training the comparison reading group offers control or equivalence. The two groups are  different in positive expectations and attention and support that are likely to be reflected the parent and teacher ratings. There’s a high likelihood of any differences in outcomes being nonspecific and not something active and distinct ingredient of mindfulness training. In any comparison with the students assigned to reading time, students assigned to mindfulness training have the benefit of any active ingredient it might have, as well as any nonspecific, placebo ingredients.

This is exceedingly weak design, but one that dominates evaluations of mindfulness.

With only 32 students per group, note too that this is a seriously underpowered study. It has less than a 50% probability of detecting a moderate sized effect if one is present. And because of the larger effect size needed to achieve statistical significance with such a small sample size, and statistically significant effects will be large, even if unlikely to replicate in a larger sample. That is the paradox of low sample size we need to understand in these situations.

Not surprisingly, there were no differences between the mindfulness and reading control groups on any outcomes variable, whether rated by parents or teachers. Nonetheless, the authors rescued their claims for an effective intervention with:

However, as shown by the significance of interaction terms, baseline levels of EF (GEC reported by teachers) moderated improvement in posttest EF for those children in the MAPs group compared to children in the control group. That is, on the teacher BRIEF, children with poorer initial EF (higher scores on BRIEF) who went through MAPs training showed improved EF subsequent to the training (indicated by lower GEC scores at posttest) compared to controls.

Similar claims were made about parent ratings. But let’s look at figure 3 depicting post-test scores. These are from the teachers, but results for the parent ratings are essentially the same.

teacher BRIEF quartiles

Note the odd scaling of the X axis. The data are divided into four quartiles and then the middle half is collapsed so that there are three data points. I’m curious about what is being hidden. Even with the sleight-of-hand, it appears that scores for the intervention and control groups are identical except for the top quartile. It appears that just a couple of students in the control group are accounting for any appearance of a difference. But keep in mind that the upper quartile is only a matter of eight students in each group.

This scatter plot is further revealing:

teacher BRIEF

It appears that the differences that are limited to the upper quartile are due to a couple of outlier control students. Without them, even the post-hoc differences that were found in the upper quartile between intervention control groups would likely disappear.

Basically what we are seeing is that most students do not show any benefit whatsoever from mindfulness training over being in a reading group. It’s not surprising that students who were not particularly elevated on the variables of interest do not register an effect. That’s a common ceiling effect in such universally delivered interventions in general population samples

Essentially, if we focus on the designated outcome variables, we are wasting the students’ time as well as that of the staff. Think of what could be done if the same resources could be applied in more effective ways. There are a couple of students in in this study were outliers with low executive function. We don’t know how else they otherwise differ.Neither in the study, nor in the validation of these measures is much attention given to their discriminant validity, i.e., what variables influence the ratings that shouldn’t. I suspect strongly that there are global, nonspecific aspects to both parent and teacher ratings such that they are influenced by the other aspects of these couple of students’ engagement with their classroom environment, and perhaps other environments.

I see little basis for the authors’ self-congratulatory conclusion:

The present findings suggest that mindfulness introduced in a general  education setting is particularly beneficial for children with EF difficulties.

And

Introduction of these types of awareness practices in elementary education may prove to be a viable and cost-effective way to improve EF processes in general, and perhaps specifically in children with EF difficulties, and thus enhance young children’s socio-emotional, cognitive, and academic development.

Maybe the authors stared with this conviction and it was unshaken by disappointing findings.

Or the statement made in Mindfulness Nation:

What is of particular interest is that those with the lowest levels of executive control73 and emotional stability74 are likely to benefit most from mindfulness training.

But we have another study that is cited for this statement.

74. Huppert FA, Johnson DM. A controlled trial of mindfulness training in schools: The importance of practice for an impact on wellbeing. The Journal of Positive Psychology. 2010; 5(4):264-274.

The first author, Felicia Huppert is a  Founder and Director – Well-being Institute and Emeritus Professor of Psychology at University of Cambridge, as well as a member of the academic staff of the Institute for Positive Psychology and Education of the Australian Catholic University.

This study involved 173 14- and 15- year old  boys from a private Catholic school.

The Journal of Positive Psychology is not known for its high methodological standards. A look at its editorial board suggests a high likelihood that manuscripts submitted will be reviewed by sympathetic reviewers publishing their own methodologically flawed studies, often with results in support of undeclared conflicts of interest.

The mindfulness training was based on the program developed by Kabat-Zinn and colleagues at the University of Massachusetts Medical School (Kabat-Zinn, 2003). It comprised four 40 minute classes, one per week, which presented the principles and practice of mindfulness meditation. The mindfulness classes covered the concepts of awareness and acceptance, and the mindfulness practices included bodily awareness of contact points, mindfulness of breathing and finding an anchor point, awareness of sounds, understanding the transient nature of thoughts, and walking meditation. The mindfulness practices were built up progressively, with a new element being introduced each week. In some classes, a video clip was shown to highlight the practical value of mindful awareness (e.g. “The Last Samurai”, “Losing It”). Students in the mindfulness condition were also provided with a specially designed CD, containing three 8-minute audio files of mindfulness exercises to be used outside the classroom. These audio files reflected the progressive aspects of training which the students were receiving in class. Students were encouraged to undertake daily practice by listening to the appropriate audio files. During the 4-week training period, students in the control classes attended their normal religious studies lessons.

A total of 155 participants had complete data at baseline and 134 at follow-up (78 in the mindfulness and 56 in the control condition). Any student who had missing data are at either time point was simply dropped from the analysis. The effects of this statistical decison are difficult to track in the paper. Regardless, there was a lack of any difference between intervention and control group and any of a host of outcome variables, with none designated as primary outcome.

Actual practicing of mindfulness by students was inconsistent.

One third of the group (33%) practised at least three times a week, 34.8% practised more than once but less than three times a week, and 32.7% practised once a week or less (of whom 7 respondents, 8.4%, reported no practice at all). Only two students reported practicing daily. The practice variable ranged from 0 to 28 (number of days of practice over four weeks). The practice variable was found to be highly skewed, with 79% of the sample obtaining a score of 14 or less (skewness = 0.68, standard error of skewness = 0.25).

The authors rescue their claim of a significant effect for the mindfulness intervention with highly complex multivariate analyses with multiple control variables in which outcomes within-group effects for students assigned to mindfulness  were related to the extent of students actually practicing mindfulness. Without controlling for the numerous (and post-hoc) multiple comparisons, results were still largely nonsignificant.

One simple conclusion that can be drawn is that despite a lot of encouragement, there was little actual practice of mindfulness by the relatively well-off students in a relatively highly resourced school setting. We could expect results to improve with wider dissemination to schools with less resources and less privileged students.

The authors conclude:

The main finding of this study was a significant improvement on measures of mindfulness and psychological well-being related to the degree of individual practice undertaken outside the classroom.

Recall that Mindful Nation cited the study in the following context:

What is of particular interest is that those with the lowest levels of executive control73 and emotional stability74 are likely to benefit most from mindfulness training.

These are two methodologically weak studies with largely null findings. They are hardly the basis for launching a national policy implementing universal mindfulness in the schools.

As noted in the US AHRQ report, despite a huge number of studies of mindfulness having been conducted, few involved a test with an adequate control group, and so there’s little evidence that mindfulness has any advantage over any active treatment. Neither of these studies disturbed that conclusion, although they are spun both in the original studies and in the Mindful Nation report to be positive. Both papers were published in journals where the reviewers were likely to be overly sympathetic and not at him tentative to serious methodological and statistical problems.

The committee writing Mindful Nation arrived at conclusions consistent with their prior enthusiasm for mindfulness and their vested interest in it. They sorted through evidence to find what supported their pre-existing assumptions.

Like UK resilience programs, the recommendations of Mindful Nation put considerable resources in the delivery of services to a large population and likely to have the threshold of need to register a socially in clinically significant effect. On a population level, results of the implementation are doomed to fall short of its claims. Those many fewer students in need more timely, intensive, and tailored services are left underserved. Their presence is ignored or, worse, invoked to justify the delivery of services to the larger group, with the needy students not benefiting.

In this blog post, I mainly focused on two methodologically poor studies. But for the selection of these particular studies, I depended on the search of the authors of Mindful Nation and the emphasis that were given to these two studies for some sweeping claims in the report. I will continue to be writing about the recommendations of Mindful Nation. I welcome reader feedback, particularly from readers whose enthusiasm for mindfulness is offended. But I urge them not simply to go to Google and cherry pick an isolated study and ask me to refute its claims.

Rather, we need to pay attention to the larger literature concerning mindfulness, its serious methodological problems, and the sociopolitical forces and vested interests that preserve a strong confirmation bias, both in the “scientific” literature and its echoing in documents like Mindful Nation.