- “Inside the Black Box” by Blue Square Thing
While scientists generally have trust in peer-review, the details of the process are largely hidden from the public, and readers are only privy to the final product.
The European Molecular Biology Organization (EMBO) wants to change that. Beginning this month, all four EMBO journals will be publishing the complete details of the peer-review process alongside the final paper. These details, packaged together as a “peer-review process file,” will contain the referee comments and author’s responses from every round of revisions, plus the editorial decision letters and a detailed timeline of each step.
Bernd Pulverer, head of scientific publications for EMBO, provides the rationale behind the changes in his article, “Transparency showcases strength of peer review,” which appeared in the November 4th issue of the journal, Nature.
For Pulverer, prying open the black box of peer-review to reveal how the system works conveys trust in the process:
In our view, these augmented papers are testament to the fact that carefully administered peer review works — works well, in fact.
In addition, data on the fate of rejected manuscripts from the EMBO Journal suggests that editorial decisions were, in Pulverer’s words, “generally informed and fair.” The vast majority of rejected manuscripts were eventually published in journals of lower impact and received fewer citations than the average article published in the EMBO Journal.
The addition of peer-review process files has had little effect on author behavior. Pulverer reports that the submission rate to the EMBO Journal has remained steady and only 5% of authors have opted-out of the process. Their reasons?
The objectors cite a reluctance to add to the already excessive literature or a perception that an otherwise excellent piece of work can be marred by prominent comments on small mistakes or limitations.
The airing of, as one referee described “the ‘dirty washing’ leading up to acceptance,” costs the journal more than just potential staining of its brand. Creating the peer-review process files increases the time and effort of authors, reviewers, and editors; the latter are responsible for removing names, correcting typos, and removing data when they are used to assuage a reviewer’s concern. The result of these additional steps results in no change in the quality of the review. If anything, it only slows the process down and makes it more expensive:
Nor have we seen a significant change in the quality of referees’ reports or authors’ responses — for better or for worse. Several referees have acknowledged that they spend more time on phrasing their reports now […] And we estimate that each file takes around 1½ hours for our administrators and editors to produce.
The move to add a peer-review process file seems to go counter to recent developments in journal publishing, which have been aiming to reduce costs and speed production through automation and by minimizing the duplication of effort.
For example, to reduce manuscript processing and markup by journal staff, the Optics Society of America requires authors to use their manuscript preparation templates. The Journal of Neuroscience has ended the practice of publishing supplemental files, citing that the process overwhelms reviewers and slows down the publication process. To reduce the burden on reviewers and speed up publication, the Neuroscience Peer Review Consortium facilitates the transfer of manuscripts and reviews among participating journals. Many publishers also assist in the transfer of files and resubmission to related journals within their portfolio (aka cascading peer review). The addition of EMBO’s peer-review process files is a counter-example to this general trend.
A report, commissioned by JISC-collections, estimates the cost of in-kind contributions to journal publications through the peer-review process to be between £110-165 million for UK researchers. While increases in academic time are largely borne by the universities who support them, increases on the editorial and publication side will ultimately be transferred to libraries through rising subscription prices, something that librarians are unilaterally against, as echoed in a recent piece in the Times Higher Education.
In the case of EMBO Journal, opening the black box of peer-review has revealed that their system was working fairly. Keeping the box open, however, will not come cheaply.
Transparency will come with a price.
Related articles
- Are Peer-Reviewers Overloaded? Or Are Their Incentives Misaligned? (scholarlykitchen.sspnet.org)
- The “Burden” of Peer-Review (scholarlykitchen.sspnet.org)
- A trip through the peer review sausage grinder (arstechnica.com)
Discussion
10 Thoughts on "The Price of Transparency and Peer Review"
Interesting post. I think, though, you are not comparing like with like when you compare the process file with the “transfer of submissions” procedure; or the process file with the “SI” now no longer being published by J Neurosci.
In the first case, EMBO J publishes the referees’ reports plus editor/author responses. (This probably is not very labour intensive.) The “transfer of submission” process occurs when a manuscript is rejected from one journal but the author transfers submission to another (eg by the same publisher or within the neuroscience consortium). The two processes are independent.
In the case of the “SI”, the “process file” is the referee reports and the author/editor responses, whereas the SI consists of additional material relevant to the research – additional figures, controls, methods, references, etc). Again, the two are independent (eg a journal could decide not to publish SI but to publish process files!).
I agree, these practices are different, and yet they are all examples of publishers looking for ways to reduce manual labor and speed up publication.
Fully open peer review has been conducted by some of the BMC series journals for years, even with the names of the reviewers shown. I like this concept as it reduces the possibility for unfair and personally offending comments sometimes found in the anonymous reviews (“I hate your paper”). It also shows the tedious efforts of the peer review process for both authors and reviewers, e.g. http://www.biomedcentral.com/1471-2407/10/266/prepub
It’s an interesting experiment. The question to me will be long-term reader interest. There’s some novelty value right now in getting to see this information. If it becomes a standard practice, in 5 years, how many readers will regularly read the reviews that come with the paper?
The real value, as the Nature article points out, is perhaps in negative reviews for rejected papers. Not sure how much you’d get out of a lot of positive reviews for accepted papers. Does it really matter to the reader that figure 6 was added at the request of a reviewer instead of coming in with the original manuscript? But as the article also points out, there’s no obvious mechanism for making the rejection information available, and authors would likely strenuously object.
I should think, with the sophisticated editorial management systems now being used, the extra cost of “publishing” the comments and responses along with the paper would be minimal. The identities of the reviewers could be kept in fields separate from the reports themselves, if there is still a desire to retain their anonymity, for example. I share the skepticism about how valuable this additional information will be to readers, however. But though this experiment apparently didn’t have this result, I could imagine some reviewers taking a bit more care to write coherently, clearly, and grammatically when they know their comments will be made public also, and this would be a benefit to the process overall.
