An old saw, borrowing from Churchill’s quip about democracy, is that peer review is the worst system for validating research except for all the others. Another effort to improve this much-maligned process has been announced. eLife, BioMed Central (BMC), the Public Library of Science (PLoS), and the European Molecular Biology Organization (EMBO) will be forming a new peer review consortium based around the concept of what eLife calls “portable peer review.”
As the Economist recently noted, this is the latest in a series of schemes, along with Rubriq and Peerage of Science, to detach referee reports from publication in a specific journal. (Readers of this article may be interested in the recent interview I conducted with Rubriq’s Keith Collier on this topic). Unlike Rubriq and Peerage of Science, which are attempting to build extra-journal validation, this initiative builds on the existing journal peer review process.
Authors who submit to a participating journal in the consortium, and are not accepted by that journal, will be able to redirect their paper, with the referee’s reports, to any other journal in the consortium. Referees will be given the opportunity to opt out of having their reports forwarded, or to forward them anonymously (in all cases, the referee’s identity will be anonymous to the author – referees will choose whether they wish to remain anonymous to the editors of the secondary journal).
“How the referee reports are used is up to the receiving journal,” noted Margaret Winker, Senior Research Editor at PLoS Medicine. “Editors may choose to use them or not. Anonymous reviews may be of interest to the editor but would not be as useful since the editor needs to know the identify of the reviewers to ascertain his or her area of expertise.”
Cascading peer review systems, of which this is a variant, have been in use for some time. Publishers with numerous journals in the same field often provide authors with the opportunity to resubmit a paper to other journals managed by the publisher without needing to go through the entire submission and review process all over again. This saves time for the author who does not need to resubmit from scratch to a new journal. It also saves time for reviewers, at least in aggregate, as there will be fewer total reviews requested in the universe.
In addition to saving time for everyone involved, cascading peer review systems can provide a competitive advantage to publishers that employ them effectively. Top-tier journals can attract more high-caliber papers by offering authors the reassurance of the possibility of a back-up plan via publishing in a second choice journal by the same publisher. Publishers including Nature Publishing Group, PLoS, BMC, EMBO, the American Medical Association, and others have used cascading peer review successfully for years.
Second- and third-tier journals benefit as well as they can receive papers, cascaded from their top-tier siblings, they would not be likely to receive as direct submissions. The authors, in this later scenario, may prefer to get a decision letter quickly with a second tier journal as opposed to starting the process all over again with a top tier journal, risking finding themselves in the same predicament several months down the road.
In many ways, the eLife/BMC/PLoS/EMBO consortium (which needs a pithy acronym) is similar to intra-publisher cascade systems, only on a larger scale. The aims of the participants are also similar.
“We all want to ensure that this peer-reviewed ‘sound science’ is published, incorporating any necessary revisions, as rapidly as possible and without duplicating the peer review effort,” said Matthew Cockerill, Managing Director of BMC. “Duplicated review effort reduces academic productivity and so effectively makes a funder’s investment in research less effective.”
Bernd Pulverer, Chief Editor of the EMBO Journal expressed a similar perspective. “Often, a manuscript may be rejected from a journal, not for reasons of quality, but rather of scope,” said Pulverer. “For example, whereas all these journals will select for conceptual advance, eLife may emphasize the broad interest of a discovery, the EMBO Journal mechanism, and PLoS Medicine human relevance.”
The eLife/BMC/PLoS/EMBO consortium is not, of course, the first multi-publisher consortium. The most notable predecessor is the Neuroscience Peer Review Consortium. While still operational, the Neuroscience Peer Review Consortium has not lived up to expectations with very few papers being referred amongst the participating journals.
I asked Pulverer why he thinks this new consortium will succeed where the Neuroscience Peer Review Consortium has not.
“We aim for the cooperation to be much more tight-knit,” replied Pulverer. “For example, if we receive a paper, we will look at the referee reports and then go back to the authors rapidly with a very clear picture of what our requirements will be. We might be able to tell the author that we will accept the paper with a subset of revisions requested by the referees or we might say that it is in the authors’ interest that we ask for input from one additional referee. In any event, the authors will have a clear picture of how we will proceed and they can chose to transfer based on that.”
Cockerill brings a different perspective. He thinks the main problem with the Neuroscience Peer Review Consortium is that the participating publishers have too much in common. “With the Neuroscience Peer Review Consortium, the strong competition between different journals in the field doesn’t create sufficiently strong incentives for the journals to make the effort to facilitate the free flow of manuscripts and reviews.” The new consortium, by contrast, while broadly centered on the life sciences, is less closely focused on a single specialty.
Time will tell whether this new consortium will be successful. And indeed numerous challenges still must be worked out, including:
- Technology Interoperability. The publishers in the consortium are not all using the same manuscript submission systems, which means that submitted papers, and their accompanying reviews, will need to be manually exported from one system and re-imported to another. This is a cumbersome process that will increase workloads at editorial offices and will need to be streamlined over time.
- Participation of Referees. The consortium will only be effective enough to be worth the trouble if referees agree to have both their reports and their identities shared amongst participating publishers in the consortium. Reviews by anonymous (to the editor/publisher) referees are of limited value.
- Editorial Participation. Just about every editor I know believes the nuances of his or her peer review process are superior to all others. Most editors would use another’s editor’s toothbrush before accepting the output of his or her peer review process. Gaining participation of editors may be the most difficult challenge of all.
If the new consortium can surmount these obstacles, however, it may indeed make life easier for authors and peer reviewers. This would be a much-welcomed outcome. It may also provide a competitive advantage to the participating publishers. It is an axiom that authors wish to be published in the journal in their field with the highest profile that they can get their paper into as quickly as possible. The ability to start with a high profile title such as PLoS Biology and quickly cascade to an alternate publication venue across four publishers removes much of the risk of submitting to such a journal. This will raise the selectivity of the flagship titles for each publisher even further while simultaneously increasing the flow of high quality papers to these same publishers’ other titles.
If this initiative is successful, other publishers will no doubt wish to join the consortia or, alternatively, to imitate it by forming additional consortia. Perhaps we will see the rise competing consortia, with complex and shifting alliances in the mode of a George R.R. Martin story. This is, perhaps, but the first move in the Game of Papers.
Note: Thanks to B.A and R.F. for background information related to this article. And thanks to Margaret Winker, Matthew Cockerill, and Bernd Pulverer for their time and interview responses.