Sordid tale of a study of cognitive behavioral therapy for schizophrenia gone bad

What motivates someone to publish that paper without checking it? Laziness? Naivety? Greed? Now that’s one to ponder. – Neuroskeptic, Science needs vigilantes.

feared_and_hated_by_a_world_they_have_sworn_to_pro_by_itomibhaa-d4kx9bd.pngWe need to

  • Make the world safe for post-publication peer review (PPR) commentary.
  • Ensure appropriate rewards for those who do it.
  • Take action against those who try to make life unpleasant for those who are toil hard for a scientific literature that is more trustworthy.

In this issue of Mind the Brain, I set the stage for my teaming up with Magneto to bring some bullies to justice.

The background tale of a modest study of cognitive behavior therapy (CBT) for patients with schizophrenia has been told in bits and pieces elsewhere.

The story at first looked like it was heading for a positive outcome more worthy of a blog post than the shortcomings of a study in an obscure journal. The tale would go

A group organized on the internet called attention to serious flaws in the reporting of a study. We then witnessed the self-correcting of science in action.

If only this story was complete and accurately described scientific publishing today

Daniel Lakens’ blog post, How a Twitter HIBAR [Had I Been A Reviewer] ends up as a published letter to the editor recounts the story beginning with expressions of puzzlement and skepticism on Twitter.

Gross errors were made in a table and a figure. These were bad enough in themselves, but seemed to point to reported results not seem supporting the claims made in the article.

A Swedish lecturer blogged Through the looking glass into an oddly analyzed clinical paper .

Some of those involved in the Twitter exchange banded together in writing a letter to the editor.

Smits, T., Lakens, D., Ritchie, S. J., & Laws, K. R. (2014). Statistical errors and omissions in a trial of cognitive behavior techniques for psychosis: commentary on Turkington et al. The Journal of Nervous and Mental Disease, 202(7), 566.

Lakens explained in his blog

Now I understand that getting criticism on your work is never fun. In my personal experience, it very often takes a dinner conversation with my wife before I’m convinced that if people took the effort to criticize my work, there must be something that can be improved. What I like about this commentary is that is shows how Twitter is making post-publication reviews possible. It’s easy to get in contact with other researchers to discuss any concerns you might have (as Keith did in his first Tweet). Note that I have never met any of my co-authors in real life, demonstrating how Twitter can greatly extend your network and allows you to meet interesting and smart people who share your interests. Twitter provides a first test bed for your criticisms to see if they hold up (or if the problem lies in your own interpretation), and if a criticism is widely shared, can make it fun to actually take the effort to do something about a paper that contains errors.

Furthermore,

It might be slightly weird that Tim, Stuart, and myself publish a comment in the Journal of Nervous and Mental Disease, a journal I guess none of us has ever read before. It also shows how Twitter extends the boundaries between scientific disciplines. This can bring new insights about reporting standards  from one discipline to the next. Perhaps our comment has made researchers, reviewers, and editors who do research on cognitive behavioral therapy aware of the need to make sure they raise the bar on how they report statistics (if only so pesky researchers on Twitter leave you alone!). I think this would be great, and I can’t wait until researchers from another discipline point out statistical errors in my own articles that I and my closer peers did not recognize, because anything that improves the way we do science (such as Twitter!) is a good thing.

Hindsight: If the internet group had been the original reviewers of the article…

The letter was low key and calmly pointed out obvious errors. You can see it here. Tim Smit’s blog Don’t get all psychotic on this paper: Had I (or we) Been A Reviewer (HIBAR) describes what had to be left out to keep within the word limit.

the actual table originalTable 2 had lots of problems –

  • The confidence intervals were suspiciously wide.
  • The effect sizes seemed too large for what the modest sample size should yield.
  • The table was inconsistent with information in the abstract.
  • Neither they table nor the accompanying text had any test of significance nor reporting of means and standard deviations.
  • Confidence intervals for two different outcomes were identical, yet one had the same value for its effect size as its lower bound.

Figure 5 Click to Enlarge

Figure 5 was missing labels and definitions on both axes, rendering it uninterpretable. Duh?

The authors of the letter were behaving like a blue helmeted international peacekeeping force, not warriors attacking bad science.

peacekeepersBut you don’t send peacekeeping troops into an active war zone.

In making recommendations, the Internet group did politely introduce the R word:

We believe the above concerns mandate either an extensive correction, or perhaps a retraction, of the article by Turkington et al. (2014). At the very least, the authors should reanalyze their data and report the findings in a transparent and accurate manner.

Fair enough, but I doubt the authors of the letter appreciated how upsetting this reasonable advice was or anticipated what reaction would be coming.

A response from an author of the article and a late night challenge to debate

The first author of the article published a reply

Turkington, D. (2014). The reporting of confidence intervals in exploratory clinical trials and professional insecurity: a response to Ritchie et al. The Journal of Nervous and Mental Disease, 202(7), 567.

He seemed to claim to re-examine the study data and

  • The findings were accurately reported.
  • A table of means and standard deviations was unnecessary because of the comprehensive reporting of confidence intervals and p-values in the article.
  • The missing details from the figure were self-evident.

The group who had assembled on the internet was not satisfied. An email exchange with Turkington and the editor of the journal confirmed that Turkington had not actually re-examined the raw file data, but only a summary with statistical tables.

The group requested the raw data. In a subsequent letter to the editor, they would describe Turkington as timely the providing the data, but the exchange between them was anything but cordial. Turkington at first balked, saying that the data were not readily available because the statistician had retired. He nonetheless eventually provided the data, but not before first sending off a snotty email –

Click to Enlarge
Click to Enlarge

Tim Smit declined:

Dear Douglas,

Thanks for providing the available data as quick as possible. Based on this and the tables in the article, we will try to reconstruct the analysis and evaluate our concerns with it.

With regard to your recent invitation to “slaughter” me at Newcastle University, I politely want to decline that invitation. I did not have any personal issue in mind when initiating the comment on your article, so a personal attack is the least of my priorities. It is just from a scientific perspective (but an outsider to the research topic) that I was very confused/astonished about the lack of reporting precision and what appears to be statistical errors. So, if our re-analysis confirms that first perception, then I am of course willing to accept your invitation at Newcastle university to elaborate on proper methodology in intervention studies, since science ranks among the highest of my priorities.

Best regards,

Tim Smits

When I later learned of this email exchange, I wrote to Turkington and offered to go to Newcastle to debate either as Tim Smits’ second or to come alone. Turkington asked me to submit my CV to show that I wasn’t a crank. I complied, but he has yet to accept my offer.

A reanalysis of the data and a new table

Smits, T., Lakens, D., Ritchie, S. J., & Laws, K. R. (2015). Correcting Errors in Turkington et al.(2014): Taking Criticism Seriously. The Journal of nervous and mental disease, 203(4), 302-303.

The group reanalyzed the data and the title of their report leaked some frustration.

We confirmed that all the errors identified by Smits et al. (2014) were indeed errors. In addition, we observed that the reported effect sizes in Turkington et al. (2014) were incorrect by a considerable margin. To correct these errors, Table 2 and all the figures in Turkington et al. (2014) need to be changed.

The sentence in the Abstract where effect sizes are specified needs to be rewritten.

A revised table based on their reanalyses was included:

new tableGiven the recommendation of their first letter was apparently dismissed –

To conclude, our recommendation for the Journal and the authors would now be to acknowledge that there are clear errors in the original Turkington et al. (2014) article and either accept our corrections or publish their own corrigendum. Moreover, we urge authors, editors, and reviewers to be rigorous in their research and reviewing, while at the same time being eager to reflect on and scrutinize their own research when colleagues point out potential errors. It is clear that the authors and editors should have taken more care when checking the validity of our criticisms. The fact that a rejoinder with the title “A Response to Ritchie et al. [sic]” was accepted for publication in reply to a letter by Smits et al. (2014) gives the impression that our commentary did not receive the attention it deserved. If we want science to be self-correcting, it is important that we follow ethical guidelines when substantial errors in the published literature are identified.

Sound and fury signifying nothing

Publication of their letter was accompanied by a blustery commentary from the statistical editor for the journal full of innuendo and pomposity.

quote-a-harmless-hilarity-and-a-buoyant-cheerfulness-are-not-infrequent-concomitants-of-genius-and-we-charles-caleb-colton-294969

Cicchetti, D. V. (2015). Cognitive Behavioral Techniques for Psychosis: A Biostatistician’s Perspective. The Journal of Nervous and Mental Disease, 203(4), 304-305.

He suggested that the team assembled on the internet

reanalyzed the data of Turkington et al. on the basis that it contained some serious errors that needed to be corrected. They also reported that the statistic that Turkington et al. had used to assess effect sizes (ESs) was an inappropriate metric.

Well, did Turkington’s table contain errors and was the metric inappropriate? If so, was a formal correction or even retraction needed? Cicchetti reproduced the internet groups’ table, but did not immediately offer his opinion. So, the uncorrected article stands as published. Interested persons downloading it from behind the journal’s paywall won’t be alerted to the controversy.

hello potInstead of dealing with the issues at hand, Cicchetti launched into an irrelevant lecture about Jacob Cohen’s arbitrary designation of effect sizes as small, medium, or large. Anything he said had already appeared clearer and more accurately in an article by Daniel Laken, one of the internet group authors. Cicchetti cited that article, but only as a basis for libeling the open access journal in which it appeared.

To be perfectly candid, the reader needs to be informed that the journal that published the Lakens (2013) article, Frontiers in Psychology, is one of an increasing number of journals that charge exorbitant publication fees in exchange for free open access to published articles. Some of the author costs are used to pay reviewers, causing one to question whether the process is always unbiased, as is the desideratum. For further information, the reader is referred to the following Web site: http://www.frontiersin.org/Psychology/fees.

love pomposityCicchetti further chastised the internet group for disrespecting the saints of power analysis.

As an additional comment, the stellar contributions of Helena Kraemer and Sue Thiemann (1987) were noticeable by their very absence in the Smits et al. critique. The authors, although genuinely acknowledging the lasting contributions of Jacob Cohen to our understanding of ES and power analysis, sought to simplify the entire enterprise

Jacob Cohen is dead and cannot speak. But good Queen Mother Helena is very much alive and would surely object to being drawn into this nonsense. I encourage Cicchetti to ask what she thinks.

Ah, but what about the table based on the re-analyses of the internet group that Cicchetti had reproduced?

The reader should also be advised that this comment rests upon the assumption that the revised data analyses are indeed accurate because I was not privy to the original data.

Actually, when Turkington sent the internet group the study data, he included Cicchetti in the email.

The internet group experienced one more indignity from the journal that they had politely tried to correct. They had reproduced Turkington’s original table in their letter. The journal sent them an invoice for 106 euros because the table was copyrighted. It took a long email exchange before this billing was rescinded.

Science Needs Vigilantes

Imagine a world where we no longer depend on a few cronies of an editor to decide once and forever the value of a paper. This would replace the present order in which much of the scientific literature is untrustworthy, where novelty and sheer outrageousness of claims are valued over robustness.

Imagine we have constructed a world where post publication commentary is welcomed and valued. Data are freely available for reanalysis and the rewards are there for performing those re-analyses.

We clearly are not there yet and certainly not with this flawed article. The sequence of events that I have described has so far not produced a correction of a paper. As it stands, the paper concludes that nurses can and should be given a brief training that will allow them to effectively treat patients with severe and chronic mental disorder. This paper encourages actions that may put such patients and society at risk because of ineffectual and neglectful treatment.

The authors of the original paper and the editor responded with dismissal of the criticisms, ridicule, and, the editor at least, libeling open access journals. Obviously, we have not reached the point at which those willing to re-examine and if necessary, re-analyze data, are appropriately respected and protected from unfair criticism. The current system of publishing gives authors who have been questions and editors who are defensive of their work, no matter how incompetent and inept it may be, the last word. But there is always the force of social media- tweets and blogs.

The critics were actually much too kind and restrained in a critique narrowly based on re-analyses. They ignored so much about

  • The target paper as an underpowered feasibility study being passed off a source of estimates of what a sufficiently sized randomized trial would yield.
  • The continuity between the mischief done in this article with tricks and spin in the past work of the author Turkington.
  • The laughably inaccurate lecture of the editor.
  • The lowlife journal in which the article was published.

These problems deserve a more unrestrained and thorough trashing. Journals may not yet be self-correcting, but blogs can do a reasonable job of exposing bad science.

Science needs vigilantes, because of the intransigence of those pumping crap into the literature.

Coming up next

In my next issue of Mind the Brain I’m going to team up with Magneto. You may recall I previously collaborated with him and Neurocritic to scrutinize some junk science that Jim Coan and Susan Johnson had published in PLOS One. Their article crassly promoted to clinicians what they claimed was a brain-soothing couples therapy. We obtained an apology and a correction in the journal for undeclared conflict of interest.

Magneto_430But that incident left Magneto upset with me. He felt I did not give sufficient attention to the continuity between how Coan had slipped post hoc statistical manipulations in the PLOS article to get positive results and what he had done in a past paper with Richard Davison. Worse, I had tipped off Jim Coan about our checking his work. Coan launched a pre-emptive tirade against post-publication scrutiny, his now infamous Negative Psychology rant  He focused his rage on Neuroskeptic, not Neurocritic or me, but the timing was not a coincidence. He then followed up by denouncing me on Facebook as the Chopra Deepak of skepticism.

I still have not unpacked that oxymoronic statement and decided if it was a compliment.

OK, Magneto, I will be less naïve and more thorough this round. I will pass on whatever you uncover.

Check back if you just want to augment your critical appraisal skills with some unconventional ones or if you just enjoy a spectacle. If you want to arrive at your own opinions ahead of time, email Douglas Turkington douglas.turkington@ntw.nhs.uk and for a PDF of his paywalled article. Tell him I said hello. The offer of a debate still stands.

 

Advice to Junior Academics on How to Get Involved With Twitter

tweet imagesI’m not a good role model for junior academics whom I encourage to get involved with Twitter. I have been experimenting turning exchanges on Twitter or my Facebook wall into blog posts, which I increasingly turn into articles. When my articles are newly published, I promote them with the full range of social media. All this takes considerable commitment of time.

It is too early to evaluate whether this is really worth it, but so far I find it quite satisfying. Yet, most novices would consider it an unacceptable investment of their time to try to follow what I do. Many are concerned about social media consuming too much time with uncertain payoffs.

So, I turned to a more junior colleague to offer them advice. She has been quite successful getting involved in Twitter, obtaining its rewards, and not letting it consume the rest of her life. I gave her a series of questions to answer, and then invited her to provide some brief tips and tricks for junior people. Looking over her responses, I’m impressed how solid and useful the advice is.

gozdeGozde Ozakinci, PhD, is a lecturer in health psychology at the University of St Andrews, Scotland. She obtained her BA in Psychology at Bogazici University, Istanbul, her M.Sc in Health Psychology at the University College London, and her PhD at Rutgers-The State University of New Jersey, USA. Her main research interests are in emotional regulation and health behaviour change. She works with diverse group of clinical and non-clinical populations from cancer patients to medical students. She also teaches behavioural sciences to undergraduate medical students and health psychology topics to M.Sc health psychology students. When not on Twitter, she can be found doing DIY around the house, consuming coffee (preferably Turkish) and enjoying walks in Scotland (preferably not in rain). More information about her research can be found here. Twitter: @gozde786

So, how did you get past the idea that Twitter is a waste of time?

I was reluctant to get involved with Twitter, thinking it was the same as Facebook which I use mostly to keep in touch with family and friends. I thought I didn’t need another potential time-sucker social media outlet. But I quickly realized Twitter is very different – something I can get much out of professionally. I dip in and out during the day and each time I have a nugget of information that I find useful. I feel that with Twitter, my academic world expanded to include many colleagues I wouldn’t otherwise meet. I am now able to keep my finger on the academic pulse better. The information shared on Twitter is so much more current than you would find on journals or conferences.

thinker-twitterFor instance, academics I follow post their latest articles on Twitter that would otherwise probably take me months to learn about . I can then ask questions of the authors themselves and chat with them. I think we all love to talk about our work! The blog posts I find through Twitter make me feel connected to my colleagues, current issues that face us, and take part in conversations that matter to me from evaluating evidence to more general issues in higher education.

How did you take the plunge and get started on Twitter?

I got hooked on Twitter right away, when I realised that I could get access to information that I would have heard either too late or sometimes never. It was like suddenly my academic daily life became a lot bigger. I could interact with many more colleagues from all over the world on a daily basis, rather than just the people in the office or collaborators over email/meetings.

Importantly, I didn’t get discouraged when people didn’t follow me back. If I really wanted people who didn’t follow me back to comment or pay attention to something I wanted to have a conversation on, then I just added them to my tweet. The day that Clare Gerada, the past president of Royal College of General Practitioners followed me back and commented me that we had common research interests was a good day!

The other thing that helped me is that I have broad academic interests so I follow people from different backgrounds and tweet about various topics: cancer to politics. So, I’m not restricted to my own area at all. That means that many people can find something of interest in what I put out there. I think this is important.

Did you start with a clear goal?

I guess in the beginning, I didn’t have clear goals but they developed over time in a natural way:

  1. Wanting to be a part of a conversation on academic topics rather than watching people I admire from sidelines.
  2. Being a source of rigorous evidence on a variety of topics and encouraging discussion (not sure how much I manage the discussion part).
  3. Being a source of encouragement/support for early career scientists (I even got invited to a talk at another university on my health psychology career because of colleagues I met on Twitter!).

How did you get your initial selection of people to follow? 

I started checking out who followed who. Like I checked out your list! I was surprised to see how many people that I wanted to get to know academically were on Twitter. Some of them were leaders in their field. I also started following editors of journals, journals themselves, bloggers in science communication in general (Dean Burnett, Suzi Gage, etc..). I also found a wonderful group of women scientists who blogged and tweeted: Athene Donald, Dorothy Bishop and Uta Frith for instance. They became somewhat role models to me. They were good scientists who cared about women in science, not because we were women but because we did what we did well. That was very empowering to me. They also found the time to tweet and write blog posts, showing me what an important tool we have through the modern communication tools.

I also follow major source of news such as NY Times, National Public Radio and Slate that I feel many of my followers don’t follow. So if I tweet something from there, it attracts their attention as that’s a source they wouldn’t normally hear from.

Was there some trial and error for you? Moments of doubt whether it was worth it?

It was VERY slow the first 6 months to get followers and at times for no apparent reason that I could fathom, there would be periods of losing 4-5 followers in a row and stagnation. I still get that and I can’t figure out why.

I found that daily engagement with Twitter is necessary. It’s not difficult for me as it makes me feel connected to the wider academic world. But you can’t take a holiday from Twitter for a month and hope that people will still be interested in following you or you’ll find new followers upon your return.

You might ask ‘why should I care about having followers? Isn’t it all a bit vain?’. Well, I see it as having something to say and sharing it with others. I tried not to get obsessed about number of followers in the beginning (although it was hard!) as I soon realized that with daily tweets/conversations and retweets, people started to follow me anyway. But I guess, the message would be ‘don’t give up and keep tweeting and following people you’re interested in’.

Can you provide junior persons some tips and tricks for getting involved with Twitter?

Don’t just get a twitter account. USE IT! You have to engage with it before it starts to pay off. Don’t worry about how many people follow you. It takes time to establish a critical mass of followers and also a certain level of engagement with other people. Don’t give up. And don’t be shy. Think about Twitter as another dissemination tool. We are in science because we do something valuable and we need to share that knowledge.

You don’t know who to follow? Everybody knows someone on Twitter, so search for them. Once you found them start looking at their followers.

Start following those who interest you. And don’t be afraid of unfollowing them if you don’t find their tweets interesting. And don’t be discouraged if they don’t follow you back. I follow almost double the number of people I have as followers. This doesn’t bother me as I get fed by their tweets.

Initiate a conversation. If you think you have something interesting to say to the person you follow but they don’t follow you back, just tag their handle and you may get them engage in a conversation with you.

Keep in mind that social media has been rightfully called a great equalizer. So it doesn’t matter at what stage of your career you’re at. You can have a conversation with people you admire and also with people at the other end of the world whom you’ve never met.

TweetHashtagYou find something interesting that you want to share, make sure you use the hashtag associated with it. Add your own comment to the retweets.  I used to be shy to do that but it adds another dimension to the communication you want to initiate rather than just a simple retweet.

Tweet at conferences using the conference hashtag. It’s a great way of meeting people as they will pick up your tweets and you theirs. It brings an engagement with the conference that I found very refreshing.

Start reading the blogs of people who advertise theirs on Twitter. This is as good strategy for you to get to a researcher’s thinking at the time.

Personal versus professional use. I use Twitter mainly for keeping on top of my field but I also tweet about my personal interests (about 20% of the time). It’s a balance you have to find. But people usually don’t want to hear all your inane thoughts.twitter-follow-me-icon

Follow Gozde @gozde786 and Jim @CoyneoftheRealm on Twitter. Think about our differences in strategy. Check out differences in whom we follow and who follow us. Freely take suggestions for whom you should follow from our lists. Compare our tweets. What differences  are apparent in what we are trying to accomplish? What is best for you? Join in favoring or replying to our tweets. Feel free to leave comments about this blog and your experience with tweeter below.