Biomarker Porn: From Bad Science to Press Release to Praise by NIMH Director

Concluding installment of NIMH biomarker porn: Depression, daughters, and telomeres

Pioneer HPA-axis researcher Bernard “Barney” Carroll’s comment left no doubt about what he thought of the Molecular Psychiatry article I discussed in my last issue of Mind the Brain:

Where is the HPA axis dysregulation? It is mainly in the minds0@PubSubMain@NIHMS2@s@0@44595.html of the authors, in service of their desired narrative. Were basal cortisol levels increased? No. Were peak cortisol levels increased? They didn’t say. Was the cortisol increment increased? Only if we accept a p value of 0.042 with no correction for multiple comparisons. Most importantly, was the termination of the stress cortisol response impaired? No, it wasn’t (Table 3). That variable is a feature of allostasis, about which co-author Wolkowitz is well informed. Termination of the stress response is a crucial component of HPA axis regulation (see PubMed #18282566), and it was no different between the two groups. So, where’s the beef? The weakness of this report tells us not only about the authors’ standards but also about the level of editorial tradecraft on display in Molecular Psychiatry. [Hyperlink added]

You also can see my response to Professor Carroll in the comments.

I transferred  another  comment  to the blog from my Facebook wall. It gave me an opportunity to elaborate on why

we shouldn’t depend on small convenience samples to attempt to understand phenomena that must be examined in larger samples followed prospectively.

I explained

There are lots of unanswered questions about the authors’ sampling of adolescents. We don’t know what they are like when their mothers are not depressed. The young girls could also simply be reacting to environmental conditions contributing to their mother’s depression, not to their mother’s depression per se. We don’t know how representative this convenience sample is of other daughters of depressed mothers. Is it unusual or common that daughters of this age are not depressed concurrent with their mothers’ depression? What factors about the daughters, the mothers, or their circumstances determine that the mother and daughter depression does not occur at the same time? What about differences with him him dthe daughters of mothers who are prone to depression, but are not currently depressed?  We need to keep in mind that most biomarkers associated with depression are state dependent, not trait dependent. And these daughters were chosen because they are not depressed…

But with no differences in cortisol response, what are we explaining anyway?

The Molecular Psychiatry article provides an excellent opportunity to learn to spot bad

From  http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/
From http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/

science. I encourage interested readers to map what is said in that into the chart at the right.

This second installment of my two-part blog examines how the exaggerations and distortions of the article reverberate through a press release and then coverage in NIMH Director Thomas Insel’s personal blog.

The Stanford University press release headline is worthy of the trashy newspapers we find at supermarket checkouts:

Girls under stress age more rapidly, new Stanford study reveals

The press release says things that didn’t appear in the article, but echoes the distorted literature review of the article’s introduction in claiming well-established links between shortened telomeres, frequent infections in chronic disease and death that just are not there.

The girls also had telomeres that were shorter by the equivalent of six years in adults. Telomeres are caps on the ends of chromosomes. Every time a cell divides the telomeres get a little shorter. Telomere length is like a biological clock corresponding to age. Telomeres also shorten as a result of exposure to stress. Scientists have uncovered links in adults between shorter telomeres and premature death, more frequent infections and chronic diseases.

From http://news.stanford.edu/news/2014/october/telomeres-depression-girls-10-28-2014.html
From http://news.stanford.edu/news/2014/october/telomeres-depression-girls-10-28-2014.html

And the claim of “the equivalent of six years” comes from direct quote from obtained from senior author Professor Ian Gotlib.

“It’s the equivalent in adults of six years of biological aging,” Gotlib said, but “it’s not at all clear that that makes them 18, because no one has done this measurement in children.”

Dr. Gotlib  seems confused himself about what he mean by the 10 to 14-year-old girls having aged an additional six years. Does he really think that they are now 18? If so in what way? What could he possibly mean – do they look six years older than age matched controls? That would be really strange if they did.

I hope he lets us know when he figures out what he were saying, but he shouldn’t have given the statement to the Stanford press officer unless he was clear what he meant.

The press release noted that Dr. Gotlib had already moved on to intervention studies designed to prevent telomere shortening these girls.

In other studies, Gotlib and his team are examining the effectiveness of stress reduction techniques for girls. Neurofeedback and attention bias training (redirecting attention toward the positive) seem promising. Other investigators are studying techniques based on mindfulness training.

That’s a move based on speculation, if not outright science-fiction. Neurofeedback has some very preliminary evidence for effectiveness in treating current depression, but I would like to see evidence that it has any benefit for preventing depression in young persons who have never been depressed

neurofeedbackGotlib’s claims play right into popular fantasies about rigging people up with some sort of apparatus that changes their brain. But everything changes the brain, even reading this blog post. I don’t think that reading this blog post has any less evidence for preventing later depression than neurofeedback. Nonetheless, I’m hoping  that my blogging implants a healthy dose of skepticism in readers’ brains so that they are immunized against further confusion from exposure to such press releases. For an intelligent, consumer oriented discussion of neurofeedback, see Christian Jarrett’s

Read this before paying $100s for neurofeedback therapy

Attention bias training is a curious choice. It is almost as trendy as neurofeedback, but would it work?  We have the benefit of a systematic review and recent meta-analysis that suggests a lack of evidence for attention bias training in  treating depression and no evidence for preventing it. If it’s ineffectual in treating depression, how could we possibly expect it to prevent depression? Evidence please!

Let’s speculate about the implications if the authors found the cortisol differences between the daughters of the depressed mothers and daughters of controls that they had hypothesized but did not find. What then could have been done for these young girls? Note that the daughters of depressed mothers were chosen because they were functioning well, not currently depressed themselves. Just because they were different from the control girls would not necessarily indicate that any cortisol variables were in the abnormal range. Cortisol levels are not like blood pressure – we cannot specify a level below which cortisol levels have to be brought down for better health and functioning.

Note also that these daughters were selected on the basis of their mothers being depressed and that could mean the daughters themselves were facing a difficult situation. We can’t make the mother-bashing assumption that their mother’s depression was inflicting stress on them. Maybe any psychobiological stress response that was evident was due to the circumstances that led to the depression of their mother. We don’t know enough to specify what levels of cortisol variables would be optimal and consistent with good coping with the situation – we let even specify what is normal. And we don’t know how the daughters would recover from any abnormalities without formal treatment when their circumstances changed.

Bottom line is that these investigators did not get the results they hypothesized. Even if they had, results would not necessarily to lead to clinical applications.

Nonetheless, the director of NIMH saw fit to single this paper out or maybe he was just picking up on the press release.

my blogThomas Insel’s Personal Blog: Depression, Daughters, and Telomeres.

Thomas Insel’s Director’s Blog starts by acknowledging that there are no genetic or imaging markers predicting risk for depression, but research by Stanford Psychology Professor Ian Gotlib and colleagues in Molecular Psychiatry is “worth watching.”

Insel describes Gotlib’s “longitudinal” research as following depressed mothers’ early adolescent daughters.

The young girls have not yet developed depression, but 60 percent will become depressed by the age of 18.

I can find no basis in the article for Insel’s claim that Gotlib has found 60 per cent of these girls will be depressed by age 18. The estimate seems exaggerated, particularly given the case mix of mothers of these girls. It appears that some or most of the mothers were drawn from the community. We cannot expect severe course and biological correlates of depression that we would expect from a more inpatient sample.

Searching the papers coming out of this lab, I could only find one study involving a 30 month follow-up of 22 daughters of depressed mothers in the same age range as the sample in the Molecular Psychiatry article. That’s hardly a basis for the strong claim of 60% becoming depressed by 18.

Insel embellishes the importance of differences in telomere length. He perpetuates the illusion that we can be confident that differences in telomere length suggest these girls were experiencing accelerated aging and what have high risk for disease when the girls reached middle and late age. Without the backing of data from the paper or the existing literature, Insel zeros in on

Troubling early sign of risk for premature biological aging and possibly age-related chronic diseases, such as cardiovascular disease. Investigating the cause and timing of decreased telomere length—to what extent it may result from abnormalities in stress responses or is genetically influenced, for example—will be important for understanding the relationship between cellular aging, depression, and other medical conditions.

Insel ponders how such young, healthy girls could possibly show signs of aging. According to him the answer is not clear, but it might be tied to the increased stress reactivity these girls show in performing laboratory tasks.

But as Professor Caroll noted, the study just does not much evidence of “increased stress reactivity.”

neurofeedback2jpgNonetheless, Insel indicates that Gotlib’s next step is

Using neurofeedback to help these girls retrain their brain circuits and hopefully their stress responses. It will be a few years before we will know how much this intervention reduces risk for depression, but anything that prevents or slows the telomere shortening may be an early indication of success.

It’s interesting that Insel sidestepped the claim in the press release that Gotlib was trying out a cognitive behavioral intervention to affect stress reactivity. Instead he presents a fanciful notion that neural feedback will somehow retrain these girls’ brain circuits and reduce their stress response throughout their time at home and prevent them getting depressed by their mother’s depression.

Oh, if that were only so: Insel would be vindicated in his requiring for funding that researchers get down to basic mechanisms and simply bypass existing diagnoses with limited reliability, but at least some ties to patients’ verbal reports of why they are seeking treatment. In his world of science fiction, patients, or at least these young girls, which come in to have their brains retrained to forestall the telomere shortening that is threatening them not only with becoming depressed later, but with chronic diseases and middle and late life and early death.

So, let’s retrace what was said in the original Molecular Psychiatry article to what was claimed in the Stanford University press release and what was disseminated in the social media of Dr. Insel’s personal blog. Authors’ spin bad science in a peer-reviewed article. They collaborate with their university’s press relations department by providing even more exaggerated claims. And Dr. Insel’s purpose is served by simply passing them on and social media.

There’s a lot in Dr. Insel’s Personal Blog to disappoint and even outrage

  • Researchers  seeking guidance for funding priorites.
  • Clinicians in the trenches needing to do something now to deal with the symptoms and simple misery that are being presented to them.
  • Consumers looking for guidance from the Director of NIMH as to whether they should be concerned about their daughters and what they should do about it.

A lot of bad science and science fiction is being served to back up false promises about anything likely to occur in our lifetimes, if ever.

promising treatmentTaxpayers need to appreciate where Dr. Insel is taking funding of mental health with research. He will no longer fund grants that will explore different psychotherapeutic strategies for common mental health problems as they are currently understood – you know, diagnoses tied to what patients complain about. Instead he is offering a futuristic vision in which we no longer have to pay for primary care physicians or mental health clinicians spending time talking to patients about the problems in their lives. Rather, patients can bring in a saliva sample to assess the telomere length. They then can be rigged up to a videogame providing a social stress challenge. They will then be given neurofeedback and asked to provide another saliva sample. If the cortisol levels aren’t where they are supposed to be, they will come back and get some more neurofeedback and videogames.

But wait! We don’t even need to wait until people develop problems in their lives. We can start collecting spit samples when they are preteens and head off any problems developing in their life with neural feedback.

Presumably all this could be done by technicians who don’t need to be taught communication skills. And if the technicians are having problems, we can collect spit samples from them and maybe give them some neurofeedback.

Sure, mild to moderate depression in the community is a large and mixed grouping. The diagnostic category major depression loses some of its already limited reliability and validity when applied to this level of severity. But I still have a lot more confidence in this diagnosis than relying on some unproven notions about treating telomere length and cortisol parameters in people who do not currently complain about mental health or their circumstances. And the lamer notion that this can be done without any empathy or understanding.

It’s instructive to compare what Insel says in this blog post to what he recently said in another post.

He acknowledged some of the serious barriers to the development of valid, clinically useful biomarkers:

Patients with mental disorders show many biological abnormalities which distinguish them from normal volunteers; however, few of these have led to tests with clinical utility. Several reasons contribute to this delay: lack of a biological ‘gold standard’ definition of psychiatric illnesses; a profusion of statistically significant, but minimally differentiating, biological findings;‘approximate replications’ of these findings in a way that neither confirms nor refutes them; and a focus on comparing prototypical patients to healthy controls which generates differentiations with limited clinical applicability. Overcoming these hurdles will require a new approach. Rather than seek biomedical tests that can ‘diagnose’ DSM-defined disorders, the field should focus on identifying biologically homogenous subtypes that cut across phenotypic diagnosis—thereby sidestepping the issue of a gold standard.

All but the last sentence could have been part of a negative review of the Molecular Psychiatry article or the grant that provided funding for it. But the last sentence is the kind of nonsense that a director of NIMH can lay on the and research community and expect it to be reflected in their grant applications.

But just what was the theme of this other blog post from Dr. Insel? P-hacking and the crisis concerning results of biomedical research not being consistently reproducible.

The relentless quest for a significant “P” value is only one of the many problems with data analysis that could contribute to the reproducibility problem. Many mistakenly believe that “P” values convey information about the size of the difference between two groups. P values are actually only a way of estimating the likelihood that the difference you observe could have occurred by chance. In science, “significance” usually means a P value of less than 0.05 or 1 in 20, but this does not mean that the difference observed between two groups is functionally important. Perhaps the biggest problem is the tendency for scientists to report data that have been heavily processed rather than showing or explaining the details. This suggests one of the solutions for P-hacking and other problems in data analysis: provide the details, including what comparisons were planned prior to running the experiment.

Maybe because Insel is Director of NIMH, he doesn’t expect anybody to call him on the contradictions in what he is requesting. In the p-hacking blog post, he endorsed a call to action to address the problem of a lot of federal money being wasted on research that can’t lead to improvements in the health and well-being of the population because the research is simply unreliable and depends on “heavily processed” data for which investigators don’t provide the details. Yet in the Depression, Daughters, and Telomeres post he grabs an outrageous example of this being done and tells the research community he wants to see more of it.

 

porn

5 thoughts on “Biomarker Porn: From Bad Science to Press Release to Praise by NIMH Director”

  1. Hello,

    It is very useful to find critiques of depression research – as someone with a strong personal interest in the field I have often been baffled trying to wade through various research studies by obvious proponents of a particular viewpoint.

    I apologize if this is slightly off topic, but I have been unable to find any reference regarding one particular point, and am guessing you may have some knowledge on the topic:
    Have there been any honest attempts to perform double-blinded studies of the effectiveness of the “evidence-based” psychological treatment(s) recommended as effective for depression? I realize I just may not have found it, or that I am missing a concept somewhere, but:
    Could one not set up a study where participants received for example two psychotherapy sessions/week, but from different counsellors? Tell everyone they are evaluating communication techniques or something like that, and further tell the counsellors that one of them will provided “emotional support” and the other will include for example CBT, then assign some participants to two emotional support providers?
    It wouldn’t be perfect by any means but it seems to at least have a chance of looking at the placebo effect of the “evidence-based” techniques, and I am wondering why (being such a very obvious start even to a layperson) I can’t find any reference to anything like that – am I just not looking in the right place?

    The reason that I ask is that there seemss to be a lot of anti-medication fuss, whereas it would seem to me that there is undoubtedly a placebo effect in antidepressants but it seems highly likely that there is also a placebo effect involved in response to therapy (simply because of the large number of treatments influenced by the placebo effect and/or participant confidence in the effectiveness of a particular approach).
    Trying to look up the actual research on the effectiveness of CBT for depression I eventually came upon studies that BA therapy has exactly the same results … And finally followed that over into a little corner that quietly concluded that the primary factor in the effectiveness was not any different form of therapy but the warmth/connection observed in a patient’s relationship to the therapist!
    In other words, I am trying to find an honest answer to what degree has anything actually been proven beyond the general concept that for some patients a supportive environment is sufficient, some only respond when antidepressants are added, and some don’t seem to respond to anything we have at the moment. Or perhaps that there is a subset of patients whose symptoms are primarily initiated/maintained by thought patterns, and a subset whose symptoms are primarily initiated/maintained by biological factors? (I know there would be overlap and that there are various additions such as TMS etc but these things don’t seem to be such an issue of controversy).
    My concern is that I was speaking with another patient about us actually getting worse with CBT and knowing of other people with that experience who had been helped (even if not fully remitted) by medication. I can quite understand a preference not to take medication if unnecessary and that some people have had bad experiences with medication. Regardless of whether it is CBT, BA, emotional support, or hanging upside down like a bat, anything that helps any depressed person is a good thing. However there are going to be other people like us out there, and even if we are not in the majority that is still a significant number of people. If there is not an honest presentation regarding what is actually proven, I think it is likely that patients whose only knowledge is via media exposure are going to sometimes end up making choices which are not in their best interest – when they would be better off making their personal decisions based on recommendations from their specific medical consultations.
    Also, If there is a strong effect from providing a supportive environment then it might open up additional treatment possibilies able to promptly assist more people with depression. (And as I knew a doctor who could provide a more supportive environment in 2 minutes than any other physician or psychologist could in 2 hours, it would then suggest that studying how exactly people manage to do that might be a very sensible strategy which could potentially be used to train other providers).
    Sorry if I have gone on a bit, but it is so challenging for a layperson to find out exactly what the true state of knowledge is that I have a lot of questions built up! I appreciate the chance to ask them, and any answers you can provide.

    Like

  2. This comment first was posted on 31 January. Later it disappeared, so here it is again by request.

    NIMH Director Thomas Insel should stick to his knitting. He has no business setting scientific directions for the field, as with the RDoC mandates. I suppose it is an occupational hazard of administrators to prefer a top-down management style. Problem is, science is a bottom-up business. His shallow hyping of the Gotlib-Wolkowitz report is a further reason to disqualify him.

    Dr. Insel is on record with an amateurish view of how clinical science moves forward. See PubMed #22869033. When he says we should sidestep the issue of a gold standard he plainly doesn’t understand the iterative process of nosology or the related concept of convergent validity. He doesn’t seem to understand the need to advance nosology by incorporating biomarkers along with clinical symptoms in diagnostic definitions, as happened in general medicine. And he shows no understanding of the need for a Bayesian perspective on the interpretation of biomarkers as well as of definitional symptoms. When he proposes tracking biological markers across current diagnostic domains he ignores the central issue of pathophysiology. Consider where we would end up if, for instance, we lumped Cushing disease together with juvenile onset diabetes mellitus, Type II diabetes mellitus, severe psychological or physiological stress, metabolic syndrome, anorexia nervosa, and pregnancy on the basis of an abnormal glucose tolerance test, which these all can display.

    The RDoC matrix that Dr. Insel insists should be adopted in new grant proposals to NIMH is a classic product of armchair scholastic theorizing. We are already seeing researchers bend like pretzels to make their proposals and journal submissions appear RDoC-friendly. The effect is contrived, to put it mildly. We can expect that the RDoC matrix eventually will go the way of the eccentrics and epicycles of 16th Century astronomy. The pressing issue is how much damage will be done before that happens? Director Insel and his lieutenants at NIMH need to get out of the way – their presumptuousness is breathtaking.

    Like

  3. While the article is well-crafted, as a Neurofeedback provider, I found the following phrase a bit misleading: Neurofeedback has some very preliminary evidence for effectiveness in treating current depression, but I would like to see evidence that it has any benefit for preventing depression in young persons who have never been depressed.
    The link to one journal article cited did not copy here…

    Other research exists. A Neurofeedback case series published by Jonathan E. Walker, MD would not doubt be gleefully shredded as poor science, by any graduate school research class….despite impressive results after only 6 sessions, and a one year follow-up. Clearly, it needs to replicated.
    Beta Training for Drug-Resistant Depression—A New Protocol That Usually Reduces Depression and Keeps It Reduced …
    An online search will readily find a pdf.
    Beyond that, I appreciate reading you postings, and sharpening my powers of observation.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s