It has never been easier to post a comment to a scientific article. Just don’t expect an adequate reply from the author — or one at all — according to a new study of scientific comments left on BMJ articles.
The article, “Adequacy of authors’ replies to criticism raised in electronic letters to the editor: cohort study” was published on August 10th in the medical journal BMJ. Its lead author, Peter C. Gøtzsche, is director of the Nordic Cochrane Centre. Tony Delamothe and Fiona Godlee, both editors of BMJ, were co-authors.
The study reports on articles that received comments and criticisms through BMJ’s Rapid Response, a feature that allows readers to post immediate responses to an article, which are then appended to the online version of the article.
Of the 350 papers that formed their study group, 105 (30%) received substantive criticism; however, less than half of these 105 papers (45%) received any reply from the author, the researchers report. More distressingly, papers that received severe criticism — comments that may serve to invalidate the study completely — were no more likely to garner a response from the author than if the article received only minor criticism. Moreover, when authors did respond to criticism, their critics were generally unsatisfied with the response. Editors, in contrast, were much more satisfied with authors’ responses.
Explaining these differences in perspective, Gøtzsche speculates that editors’ receptive attitude may reflect a defense of their decision to publish the article and a desire to protect the image of the journal. Editors may simply be less qualified than critics to evaluate the suitability of an author’s response.
Providing a companion editorial on the article, David Schriger and Douglas Altman argue that these results provide a strong argument for the existence of an independent letters editor.
Gøtzsche et al. provide several recommendations for journal editors, among them:
- Encouraging authors to respond to criticisms or following up when responses are inadequate;
- Sending comments and replies out to when the editor lacks sufficient knowledge about the research; or
- Requiring authors to respond to any and all comments as a precondition to manuscript acceptance
Science should involve mechanisms for dialog and self-correction, writes Gøtzsche, who recommends that journal editors consider an online response feature for their journal, if they have not done so already, and not place any time restrictions on the commenting process. He writes:
Science has no “use before” date but evolves through open debate.
The benefits of implementing technology also come with costs, and there is some rationale for placing limits on letters to the editor. Many popular articles receive substantial comments, many of them laudatory, perfunctory, or serving no other purpose than to provide a forum for other researchers to self-promote. In these cases, it may be difficult for future readers to find important criticism. Allowing important voices to be heard may justify the silencing of others.
Still, most papers represent monologues rather than dialogues. The vast majority of scientific papers receive no comments. And while the number of scientific articles grows each year, the number of letters remains the same. In their editorial, Schriger and Altman write:
[A] mountain of poor quality unfocused literature has left its readership fatigued, numb, and passive. . . . Each new paper is another monologue added to the heap. Few read it and fewer care. Errors remain unnoticed or un-noted, and no one seems terribly bothered.
Schriger and Altman argue that inadequate post-publication review cannot be remedied by simply changing the mechanics of public feedback because the problem lies more deeply in the embedded cultures and reward system of researchers. Ultimately, we need a change of culture that places more value on public discussion.
Until then, post-publication review may continue to be spotty and unreliable and an inadequate substitute — although a distinct addition to — peer review.