While scientists generally have trust in peer-review, the details of the process are largely hidden from the public, and readers are only privy to the final product.
The European Molecular Biology Organization (EMBO) wants to change that. Beginning this month, all four EMBO journals will be publishing the complete details of the peer-review process alongside the final paper. These details, packaged together as a “peer-review process file,” will contain the referee comments and author’s responses from every round of revisions, plus the editorial decision letters and a detailed timeline of each step.
Bernd Pulverer, head of scientific publications for EMBO, provides the rationale behind the changes in his article, “Transparency showcases strength of peer review,” which appeared in the November 4th issue of the journal, Nature.
For Pulverer, prying open the black box of peer-review to reveal how the system works conveys trust in the process:
In our view, these augmented papers are testament to the fact that carefully administered peer review works — works well, in fact.
In addition, data on the fate of rejected manuscripts from the EMBO Journal suggests that editorial decisions were, in Pulverer’s words, “generally informed and fair.” The vast majority of rejected manuscripts were eventually published in journals of lower impact and received fewer citations than the average article published in the EMBO Journal.
The addition of peer-review process files has had little effect on author behavior. Pulverer reports that the submission rate to the EMBO Journal has remained steady and only 5% of authors have opted-out of the process. Their reasons?
The objectors cite a reluctance to add to the already excessive literature or a perception that an otherwise excellent piece of work can be marred by prominent comments on small mistakes or limitations.
The airing of, as one referee described “the ‘dirty washing’ leading up to acceptance,” costs the journal more than just potential staining of its brand. Creating the peer-review process files increases the time and effort of authors, reviewers, and editors; the latter are responsible for removing names, correcting typos, and removing data when they are used to assuage a reviewer’s concern. The result of these additional steps results in no change in the quality of the review. If anything, it only slows the process down and makes it more expensive:
Nor have we seen a significant change in the quality of referees’ reports or authors’ responses — for better or for worse. Several referees have acknowledged that they spend more time on phrasing their reports now [...] And we estimate that each file takes around 1½ hours for our administrators and editors to produce.
The move to add a peer-review process file seems to go counter to recent developments in journal publishing, which have been aiming to reduce costs and speed production through automation and by minimizing the duplication of effort.
For example, to reduce manuscript processing and markup by journal staff, the Optics Society of America requires authors to use their manuscript preparation templates. The Journal of Neuroscience has ended the practice of publishing supplemental files, citing that the process overwhelms reviewers and slows down the publication process. To reduce the burden on reviewers and speed up publication, the Neuroscience Peer Review Consortium facilitates the transfer of manuscripts and reviews among participating journals. Many publishers also assist in the transfer of files and resubmission to related journals within their portfolio (aka cascading peer review). The addition of EMBO’s peer-review process files is a counter-example to this general trend.
A report, commissioned by JISC-collections, estimates the cost of in-kind contributions to journal publications through the peer-review process to be between £110-165 million for UK researchers. While increases in academic time are largely borne by the universities who support them, increases on the editorial and publication side will ultimately be transferred to libraries through rising subscription prices, something that librarians are unilaterally against, as echoed in a recent piece in the Times Higher Education.
In the case of EMBO Journal, opening the black box of peer-review has revealed that their system was working fairly. Keeping the box open, however, will not come cheaply.
Transparency will come with a price.
- Are Peer-Reviewers Overloaded? Or Are Their Incentives Misaligned? (scholarlykitchen.sspnet.org)
- The “Burden” of Peer-Review (scholarlykitchen.sspnet.org)
- A trip through the peer review sausage grinder (arstechnica.com)