Join PubMed’s Revolution in Post Publication Peer Review

At 11 AM on October 22, 2013, the embargo was lifted and so now it can be said: PubMed Commons has been implemented on a trial basis. It could change the process of peer assessment of scientific articles forever.

PubMedSome researchers can now comment on any article indexed at PubMed and read the comments of others. It is a closed and closely watched pilot testing. Bugs may become apparent that will need to be fixed. And NIH could always pull the plug. But so many people have invested so much at this point and spent so much time thinking through all the pros and cons that this is hopefully unlikely.

The implementation could prove truly revolutionary. PubMed Commons is effectively taking post-publication peer review out of the hands of editors and putting control firmly in the hands of the consumers of the scientific literature—where it belongs.

PubMed Commons allows us to abandon a thoroughly antiquated and inadequate reliance on letters to the editor as a means of addressing the many shortcomings of pre-publication peer review.

PubMed Commons is

  • A forum for open and constructive criticism and discussion of scientific issues.
  • That will thrive with high quality interchange from the scientific community.

You can read more about the fascinating history of PubMed here.  PubMed is a free database of references and abstracts from life sciences and biomedical journals. It primarily draws on the MEDLINE database and is maintained by the US National of Library Medicine (NLM). For 16 years ending in 1997, MEDLINE had to be accessed primarily through institutional facilities like University libraries. That excluded many  who draw on PubMed today from using it.Medline287

But then in 1997, in a revolutionary move similar to the launching of PubMed Commons, PubMed made its electronic bibliographic resources free to the public. Everyone was quite nervous at the time about being shut down. Lawyers of the for-profit publishers predictably descended on NIH to try to block the free access to abstracts, arguing, among a number of other things, copyright infringement that cut into their ability to make money. But fortunately NIH held its ground and Vice President Al Gore demonstrated PubMed’s capacity in a public gore pubmed

So, in the first revolutionary move, the for-profit journals lost their control over access to abstracts. In the second move, they are losing control over post publication commentary on articles– unless they succeed in squashing PubMed Commons.

Who can participate in PubMed Commons at this time?

  • Recipients of NIH (US) or Wellcome Trust (UK) grants can go to the NCBI website and register. You need a MyNCBI account, but they are available to the general public.
  • If you are not a NIH or Wellcome Trust grant recipient, you are still eligible to participate if you are listed as an author on any publication listed in PubMed, even a letter to the editor. But you will need to be invited by somebody already signed up for participation in PubMed Commons. So, if you have a qualifying publication, you can simply get a colleague with the grant to sign up and then invite you.

Inadequacies of letters to the editor as post-publication commentary

Up until now, the main option for post publication commentary has been in later reviews of the literature, although there was a misplaced confidence in the more immediate letters to the editor.

letter-to-editor-scaled500I regret my blog post last year recommending writing conventional letters to the editor. Letters remain better than journal clubs for junior investigators eager to develop critical appraisal skills. But it could be a waste of time to send the letters off because letters are simply not effective contributions to post publication commentary. Letters never worked reliably well, and for a number reasons, they are now obsolete.

In the not-so-good-old days of exclusively print journals, there was a rationale for these journals putting limits on letters to the editor.

  • With delays in availability due to the scheduling of print journals, letters to the editor were seldom available in a timely fashion. Readers usually would have long forgotten the article being critiqued when the letter finally came out.
  • With limits on the number of pages allowed for issue, letters to the editor consumed a scarce resource without contributing to the impact factor of the Journal. So, journals typically had strict restrictions on the length of letters (usually 400 to 800 words), and a tight deadline to submit a letter after publication of the print article, usually no more than three months or less.

Editorial Review of letters to editor has seldom been fair.

  • There is a prejudice against accepting anything but the most vitally relevant commentary. Yet editors are adverse to accepting critical letters that reflect badly on their own review processes. Get past the significance criterion and then you still risk offending the editor’s sense of privilege.
  • While letters to the editor are subject to peer review, responses from authors generally are not. Authors are free to dismiss or distort any criticism of their work, sometimes with the most absurd of statements going unchallenged.
  • Electronic bibliographic resources have become the principal means of accessing articles, but no links are often provided between a letter to the editor and the target article. So, even if the credibility of the published article with thoroughly demolished in a letter to the editor, readers accessing that article through electronic bibliographic source are not informed.
  • Many journals allowed authors to veto publication of any criticism of their work,hammering3pg but the journals do not state this in their instructions to authors. You can submit a letter to the editor, only to have it rejected because the author objects to what you said. But you are told nothing except that you letter is rejected.
  • Many journals allow authors of the target articles the last word in responding to critical letters. Publishing a single letter and a response typically completes discussion of a target article. And the letter writer never gets to see the author’s response until after it is published. So, you can put incredible effort into carefully expressing your points within the limits of 400 to 800 words, only to be made to look ridiculous with mischaracterizations you can do nothing about.

Letters to the editor are thus usually untimely, overly short, and inadequately referenced. And they elicit inadequate and even hostile responses from authors, but are generally ignored by everybody else.

Letters to the editor are seldom cited and this is just one reflection of their failing to play a major role in moving the scientific discussion forward.

The advent of web-based publishing made restrictions on letters to the editor less justifiable. Once a basic structure for processing and posting letters to the editor is set up, processing and posting cost little.

Print journals can reduce costs by maintaining a separate web-based place for letters to the editor, but restrictions on length and time to response have nonetheless continued, even if their economic justification has been lost.

bmj-logo-ogBMJ Rapid Responses provides an exceptional model for post publication peer commentary. BMJ accepts electronic responses that can be accessed by readers within 72 hours, as long as the responses are not grossly irrelevant or libelous. Readers can register “likes” of Rapid Responses and threads of comments often develop. Then, a few comments are selected each week for editing and published in the print edition. Unfortunately, the rapid responses that remain only electronic are not indexed at PubMed and can only be found by going to the BMJ website, which is behind a pay wall for most articles.

Other journals are scrambling to copy and improve upon the BMJ model, but it can take some serious modification of software and that takes time. “Like turning the Titanic around” an editor of one of the largest open access journals told me.

Until such options become widely available, a reluctance to writing letters to the editor remains thoroughly justifiable. Few letters will be submitted and fewer will be published or result in a genuine scientific exchange. And the goal remains elusive of readily accessible, indexed, citable letters to the editor and comments for which writers can gain academic credit.

PLOS cofounder Michael Eisen Photo by Andy Reynolds from Mother Jones

That is, unless PubMed Commons catches on.  It provides a potential of realizing PLOS Co-founder and disruptive innovator Michael Eisen’s goal of continuous peer assessment and reassessment, not stopping with two or three people making an unreliable, but largely irreversible judgment that something should have been published and should eternally be accepted as peer-reviewed.

PubMed Commons is only a rung on the ladder towards overthrowing the now firm line between publication and peer assessment. It’s not a place to stop, but in important step. Please join in and help make it work. If you’ve ever publish an article listed in PubMed, find a way to get invited. If you not ready to post your own comments, lurk, offer others encouragement with “like’s”, and then when the spirit moves you, jump in!

I expect that someday soon you’ll be a say to more junior colleagues, “I was active in the field when authors could prevent you from commenting on their work and editors could prevent you from embarrassing them with demonstration of the stupidity of some of their decisions.” And you junior colleagues can respond “wow, was that in the days before email? Just how did you participate in the dialogue that is at the heart of scientific communication back then? Did you have to get up and challenge speakers at conferences?”


35 thoughts on “Join PubMed’s Revolution in Post Publication Peer Review”

  1. Great and exciting news!

    I actually clicked on “how do I join?” and immediately received a link to comment, without the need of being invited by a NIH/Wellcome Trust grantee. Don’t know if this is general or not.


    1. Good question. I think that PubMed has the advantage of being a heavily trafficked gateway even to authors and journals that do not especially want comments, but can’t prevent them. The comments will eventually be linked and accessible so that anyone accessing the article will be notified of the comments, even before they click to open the abstract. There is concern, however, whether senior scientists will get engaged. Some are certainly concerned about the comments that can be left about their work and may on that basis be motivated to respond.


  2. At the risk of being judged annoying, but also with great pride in my profession as a scientific editor, I have to question:

    In the phrase, “Yet editors are adverse,” did you not mean to say “editors are averse”? Adverse is typically reserved to describe conditions or situations, not people. I’m willing to admit that, like many words in our ever-changing language, perhaps its usage is changing, too–although I missed that memo.

    And why the apostrophe when describing “likes”? It’s a simple plural, is it not?

    I, like everyone nowadays, must spend most of my time writing online. However, especially in a publications-related piece, I’d expect an editor to go over my writing with a fine-toothed comb. I’m slightly disappointed that yours had not (as also evidenced by the typo in the line “It’s not a place to stop, but in important step.”)

    My intent with this comment is not to be mean-spirited, but simply to point out that, although I very much appreciate the content of your piece (and your work in general), it likely would have been made stronger by thoughtful editing.


    1. Thanks for your attention to my grammar and word choice. Perhaps if I keep at this, I will get paid and maybe even be provided a copy editor. Adverse witnesses are generally people, except in rare circumstances like the situation portrayed in the movie The Advocate. “Like’s” refer to “like” buttons and their usage and are to be distinguished from simply liking something in a comment. I recommend you take some time to read my sometimes co-author Robin Lakoff’s book, The Language Wars.


      1. Thanks for the reply! (I didn’t actually expect you to print the comment.) I see where you are coming from, but I’ll still disagree with you about the usage of “adverse” and the apostrophe in “like’s.” I’ll check out Lakoff’s book. Thanks for the recommendation.


      2. It was a fair comment. And I cringed about carelessly having put “adverse” for “averse” and had to check. I’m actually quite dyslexic and I often don’t “see” what I write when I look at the screen, even several times. I know, I’ve written hundreds of papers and coach writers in Europe, but I remain struggling with “seeing” my goofs. My reliance on voice recognition software only ensures that my substitutions generally won’t be misspellings.

        My all-time low was responding in an email to a colleague who was going to Virginia with the question “why are you getting a vagina?” Thanks to my voice recognition software.

        I’ m sure that if you follow my blog posts, you’ll see lots of typos, and feel free to point them out. Even if I cringe, better to fix them if I agree.

        Robin Lakoff’s book is an excellent assault on the language and grammar police and a fun read.


      3. Many of us suffer the same fate and only speak one language as well, so hats-off to you and your blog Your content is spectacular. I generally don’t notice anything at all other than the message, which is why I come here after all. Oh, and that voice recognition software has a wicked sense of humor.


    1. From Hilda Bastian (and I agree):

      I agree that working out how to make participation from wider perspectives is critical, and I thought I made that clear – at least, I hope I did. I personally began my involvement in science as a consumer advocate, not university-qualified in any discipline at all, so my background was totally outsider.

      That said, there is anxiety, too, you’re right. The internet is renowned for problems with trolling, and science commenting systems have a serious problem with inflammatory and libelous statements. Consider the scale of PubMed – with tens of millions of entries and millions of visitors every day – and I hope you can see why a concern about trolling is to be expected.


      1. Hilda, am not sure what you are referring to when you say you “have made it clear” – so cannot respond to that.
        I recognise that there are challenges with the online comments – but I am sure there are better options that would not limit participation to those that are in the big boys club or have a friend within it to invite them. I have many publications that are listed in pubmed, review and edit for journals that are listed in PubMed and I seem to be excluded simply because of where my funding comes from. I am sure that there is a better solution that is inclusive and not sectarian.


      2. Fabiana, consultants from various listservs and blogs left the organizers hypersensitive to the possibilities of either trolls and nastiness or expensive moderation that is not affordable at this time. The current situation is a compromise that hopefully will be changed soon. I have taken the initiative to invite foreign and junior persons. I have taken the liberty of sending you an invitation and hope you will enjoy and invite others, further breaking down the barriers to getting a full range of participation.



      3. Thanks James. (and thanks for the invite) I am not failing to understand the concerns – I think the NIH/Wellcome barrier is too high (and biased). I am not sure why other funders (eg NSF, Royal Society, etc) weren’t included in the original launch – even if that meant delaying the launch for a wee while. I do understand the need to narrow – but narrowing at that top end is what I find elitist.


      4. My understand that participation in the launch by agencies was contingent on their approving the concept and contributing. There were lots of consultations solicited and as seen in this blog, not everyone thinks the effort will succeed. and

        The NSF consistently argues it is the elite source of funding in the US and certainly many academic psychologists consider it to be so.

        How do you define “elitist”? The launch was delayed for quite awhile due to the impending shutdown of the US government. You have not made an effective case for labeling the launch “elitist” and “biased”, unless I am missing something.


      5. The reason I see it as elitist is that there is a top down decision as to who has first rights – and if first rights are given to those with funding from 2 major agencies then that biases to a relatively small proportion of scientists world-wide. In the end, whether other agencies were invited to participate or not, it still is a top-down approach, which biases by funding agency. I would have preferred to have first rights given perhaps to authors, that is those who contribute work to the PubMed database. I am not clear as to why this more inclusive approach was not an option. That would have been less exclusive, since there would be less bias to “big nation” scientists. While I appreciate your invitation to join the discussion, my participation should not need to depend on the good will of someone else. Separating people by funding (and narrowly so) rather than contribution, and that funding is limited to 2 mainly Northern Hemisphere funders is what I find “elitist” and “biased”. I am not sure how else I can put it – not having first rights (because I happen to be outside of that narrow population) and having to “ask” to participate is frustrating to me as someone that does research from another part of the world and gets funded by a different set of agencies.


    2. I can see what you mean, Fabiana, but it is a pilot: a major recruitment exercise of additional agencies, before the system was beta-tested and given the go-ahead, isn’t a simple thing. There is an option for research institutions from anywhere to provide lists. And it’s indicated that ways to join is intended to be expanded after beta-testing.


      1. Thanks Hilda for your response.

        I didn’t find that option when I tried to register here

        Thanks to your prompt I see that the option for institutions to submit lists is instead in the FAQ page. Would it be possible to add that clarification on the main “how to join page” to make people (like me) aware that there are other ways of getting in?


  3. There are a number of other interesting initiatives focused on post publication peer review. PubCommons might benefit from this prior “experiments”:
    Faculty of 1000 @
    Pubpeer @


  4. My sincere best wishes and gratitude for this new endeavor. Of necessity, peer review is made by, well, peers – fellow specialists in a given area. Almost by definition, specialists are prone to share common and often unconscious assumptions. This approach opens “review” to a larger community and, in doing so, could have very positive benefits across the entire range of research. Nothing like a little interdisciplinary review to wake up the little grey cells.


  5. I certainly hope they find a way to make this work, and open it to more people who have knowledge about a subject but are not in the grant/research/publication pathway. There will probably be those who will try to undermine this system because a wider opportunity to review will not be appreciated by those who benefit from the current system. The old boy net work does prevent some views from getting any exposure, and they will not give up this power willingly. It sets some fields back years. And then there are the out and out trolls who just enjoy disrupting and destroying. I am not sure that there is a way to stop these people without some form of moderation.


    1. Agreed, the goal is to open it up as soon as possible after demonstrating that the pilot phase does not bring the trolls out in ways that would require moderation. And we can all defeat the domination by old white males like me by inviting qualifying junior persons. women, and those from outside the UK. All invitees need is a single listing in PubMed.


  6. Thrilled by this development. Please refer to the late,if unlamented, Web Journal “Treatment “which offered post peer review as a real advantage and foundered by receiving tiny expressions of opinion that did not reach the level of reasoned discourse. (Other toxicities also.)
    I hope their develops some independent mechanism for marking certain contributions”Problematic” accompanied by a diligent statement of What supports that view.
    Naturally both should go up on the Web avoiding any hint of suppression.These may start an interesting thread.
    If such a review device could be established the question of whether the web contribution goes into your Biblio must be faced, Since publication is a major motivation for contributing to the academic free for all -probably the best answer is a flat prohibition.This would further restrict communications to those that are not confounded with academic upcrawl.
    Don Klein


    1. Don, there is a whole story to be told about the born-and-dead-before-its-time initiative of the two APAs (American Psychological Association and American Psychiatric Association) to create what was probably the first online journal Prevention and Treatment (1997-2003) in psychology and psychiatry. On the psychology side, the politics were pretty intense, and there were deliberate attempts to politicize publication in the review process, some of them quite shameful. Nonetheless, some very smart stuff got through, particularly in the early stages. It’s a pity that no one knows to look for these papers.

      Maybe some time we can collaborate and tell the story with your firsthand knowledge as having been the first editor from the psychiatry side.

      As for getting credit for contributions to PubMed Commons, I know that there has been some discussion of some comments being designated more important and worthy of citations than others. I’m a great admirer of the BMJ system by which anyone can post a rapid response to an article that isn’t libelous or plain crazy. These are accessible on the web, but once a week or so, a small number are harvested and published in the bound paper version of the journal. They are indexed and citable.

      While there were ample resources to launch PubMed Commons, not the least of which were hard-working people being willing to donate their time, there were resources to have 24/7 monitors of comment. So far so good: there hasn’t been a great threat of irrelevant comments and trolls.


  7. Post-publication peer review is alive and well at the Copernicus suite of journals (of which at least 10 are post-publication open review). I have been researching journal peer review for the past two years – as a scientific object of study. Interestingly, based on scholarship in legal studies, post-publication open peer review is the most likely to contribute to rational decision-making for referees who engage in judgement. Two relational conditions that contribute to rational decision-making are met: transparency and accountability.

    Following are a few pre-prints you may wish to explore from my research:

    Socio-historical: – Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part I. uO Research. Pp. 1-24. – Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part II. uO Research. Pp. 1-20.

    an abridged version – Gaudet, J. 2014. Journal peer review as scientific object of study. uO Research. Pp. 1-11.

    Contemporary shaping: – Gaudet, J. 2014. All that glitters is not gold: The shaping of contemporary journal peer review at scientific and medical journals. uO Research. Pp. 1-23.

    Empirically investigating purported resistance to new ideas at journal peer review: – Gaudet, J. 2014. How pre- publication journal peer review (re)produces ignorance at scientific and medical journals: a case study. uO Research. Pp. 1-67.


      1. Thank you for your interest James. I will look forward to continuing the conversation on journal peer review. Remember that my perspective is a sociological one… hence the relational lens.

        My hope is that one day we can attain a transdisciplinary and scientific approach (not based on common experience) to an object of study that we not only share across the natural and the social sciences, but that we also engage with as actors…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s