Peer review of journals has been evolving ever since it was first introduced in the seventeenth century. Today there are a multitude of peer review processes, many different flavors of review, and a wealth of new tools and services for editors and reviewers. We asked experts from three very different organizations, each with a strong commitment to peer review, to give us their thoughts on how it’s evolved in their organizations and the communities they serve, how it works today, and what it might look like in future.
Rachel Burley is Publishing Director of BioMed Central (BMC), an early and highly successful adopter of open access journals with a strong tradition of experimentation in its publishing program. Seth Denbo is Director of Scholarly Communication and Digital Initiatives at the American Historical Association, the largest professional organization serving historians. Phil Hurst and Stuart Taylor are, respectively Publisher and Publishing Director at The Royal Society, the world’s oldest learned society and publisher of the world’s first scholarly journal, Philosophical Transactions, which celebrated its 350th birthday in 2015.
How did peer review evolve in your organization/community?
Phil and Stuart: At the Royal Society in the 1660s, Henry Oldenburg became the founding editor of the world’s first and longest-running journal dedicated to science, Philosophical Transactions. He acted as both editor and publisher. Although the journal had approval of the Society, Oldenburg had independent control of its contents and finances. Much of the early copy was sourced from his Europe-wide network of ‘natural philosophers’. The work of, or sponsored by, the Society was instead published separately in books and treatises.
Under Oldenburg’s control there was little evidence of formal quality control or what we would now call peer review. Unlike the modern journal editor, Oldenburg did not sift freely submitted manuscripts and select for publication on merit. Instead, he worked hard to secure copy from his contacts and regularly extracted, summarized and translated from existing material, or reported on scientific findings second or third hand. His motivation was primarily to communicate and to make foreign material available to his British colleagues. This was in contrast with the close scrutiny required of work published by the Society, which required approval of Council.
After Oldenburg’s death, Philosophical Transactions was edited by Secretaries of the Society and much of the content originated from the organization. Under this model, the Society was vulnerable to accusations of publishing poor quality material but having little control over it. In 1752, the Society officially took over editorial and financial control in order to protect its reputation more closely. By the 1830s the process of reviewing became more formalized, involving the organized scrutiny of specialists and written reports. Referees of Philosophical Transactions were looking for plausibility, originality and significance.
During the latter part of the 20th century, in addition to these criteria, many journals started to focus on potential impact of claims in scientific articles. The ultimate examples of this practice can be seen in the so-called elite journals (such as Nature, Science and Cell) which have rejection rates well above that required to merely weed out poor research. The Journal Impact Factor was created as a measure of the average number of citations received by an article in a given journal. Initially intended only as a guide to subscribing librarians, it started to be used as a surrogate metric with which to evaluate individual researchers for the purposes of jobs and funding.
Seth: Journals that publish historical scholarship employ a range of different peer review methodologies for evaluating and developing quality, but most are double blind, so neither the author nor the reviewers are aware of the others’ identity. Peer review itself, as it is practiced in most historical journals now, has a history. When the American Historical Review (AHR) was first founded in the final decade of the nineteenth-century, “Many of the procedures subsequently taken as guarantees of scholarly gatekeeping — such as peer review — were largely absent” writes Rob Townsend in History’s Babel: Scholarship, Professionalization, and the Historical Enterprise in the United States, 1880-1940. Contributions to the AHR — one of the few journals in the field that publishes across all historical fields and time periods — in its earliest days were not subjected to peer review. Most were written by members of the editorial board or their students. It wasn’t until the 1930s, as historical scholarship became more specialized, that articles were sent to reviewers who could assess the quality from a particular subject point of view. The identity of the reviewer was often kept hidden from the author, but consistent use of blind reviewing practices didn’t become the norm until later in the century.
How does peer review work today?
Seth: Today, the AHR has a policy of double blind reviewing. Each piece is read by the in-house editorial staff (associate editor and editor), who assign it to members of the board of editors for an initial round of review. If the board of editors deem that the piece is worth putting forward it is sent to external reviewers. Once the piece leaves the hands of the in-house editorial staff all subsequent reading is done without names attached. The board member’s review and any subsequent external reviews are anonymous, with neither the authors nor the reviewers knowing each others’ identities. Peer review has a number of different functions for the journal. Alex Lichtenstein, editor of the AHR, says “Peer review serves several important purposes. Most importantly, it assures that field-based scholarship remains innovative, up to date, attentive to historiography, and rooted in deep and informed research. It also helps us direct our authors to crafting articles that speak beyond the confines of a particular field to a wider historical audience, even while significantly advancing the scholarship in the narrower case.”
It’s a collaborative process through which potential is developed into a finished scholarly work. While this exchange of ideas between scholars also performs a central role in ensuring that the journal publishes the scholarship of the highest quality, the value in peer review is as much the process as the product. Historians who have been published in the AHR generally feel that the feedback they receive is highly valuable, and plays a vital role that goes far beyond gatekeeping. The publishing process that carefully manages peer review — which aims to ensure that every piece is carefully evaluated by the right group of scholars (in the case of the AHR as many as six rounds of review) — is time-consuming and labor intensive. It is also the basis of the production of scholarship and scholarly communication.
It’s a collaborative process through which potential is developed into a finished scholarly work. While this exchange of ideas between scholars also performs a central role in ensuring that the journal publishes the scholarship of the highest quality, the value in peer review is as much the process as the product.
Rachel: Peer review is the cornerstone of maintaining quality and integrity in the scientific record and to advance discovery. If we want to be able to rely on research, some form of peer review will always be needed. And in a climate of ever-increasing research output, it is likely that the systems used by publishers will continue to evolve, perhaps more rapidly than they have done in the past.
BMC was one of the first publishers to truly open up peer review, and this year we have led on new initiatives to further develop the process, particularly in regard to improving reproducibility and reducing the bias towards positive results. If successful, these pilot projects could become standard offerings at BMC and our parent company, Springer Nature.
We are experimenting with automating wherever possible to lighten the workload of our reviewers and editors (such as with ethics or image checks) as each manuscript goes through an average of 1.5 rounds of checks per submission. We have introduced Registered Reports at BMC Biology, BMC Medicine and BMC Ecology. This is an innovative new research format which aims to minimize bias by allowing authors to submit their rationale and methods for peer-review before any experiments are conducted. Articles that pass peer-review will be accepted-in-principle meaning that, as long as the study is completed in accordance with the pre-registered methodology, the article will be published following a second round of peer-review.
Phil and Stuart: As of June 2017, the Scopus database listed 36,377 academic journals. The great majority of them practice peer review incorporating some or all of the following elements:
- To verify the accuracy of research findings as far as possible
- To evaluate the methods and statistics
- To highlight any ethical concerns
- To make constructive criticisms aimed at improving the article
- To advise the Editor on the decision to accept/reject (in highly selective journals, this will include an estimate of potential impact/novelty etc)
- To check that the article is compliant with the data sharing policy of the journal (though this may be done by in-house staff)
Many argue that journals should not attempt to select articles based on estimates of potential impact or importance (especially given that print page limits are now largely an irrelevance and that such judgements are inevitably somewhat subjective and dependent on the research trends of the day). This has given rise to a number of journals in which peer review is more ‘objective’ i.e., based only on the soundness and rigor of the methodology. The first such journal was PLOS One (2006). The promise of this type of journal is that the scientific literature might gradually become less biased against negative or null results and will be less dominated by the trends and ‘hot topics’ of the day. At the Royal Society, nine of ten journals currently operate peer review to verify claims and to assess potential impact. Our newest journal Royal Society Open Science, however, only judges the former and leaves any assessment of impact and importance to the reader.
The time of peer reviewers is very valuable and is becoming more and more precious as research output continues to grow. Serial rejection of sound research by one journal after another wastes a considerable amount of this resource and may lead to needlessly duplicated research.
What does the future hold for peer review?
Rachel: To truly revolutionize peer review, more transformative changes will be needed. Earlier this year, we issued a report based on the discussions at the SpotOn conference in London that examines how peer review can be improved for future generations. The report offers key recommendations to the academic community that include finding and inventing new ways of identifying, verifying and inviting peer reviewers; investing in reviewer training programs, and recognizing reviewers. The report concludes that in order to affect real, industry-wide improvements, publishers, researchers, funders and institutions need to be willing to experiment with different models of peer review, particularly those that increase transparency, encourage more diversity in the reviewer pool, utilize Artificial Intelligence, and those that support training and mentoring.
Making the peer review process and experience better and more efficient is fundamental to our commitment to improve the overall publishing process.
Making the peer review process and experience better and more efficient is fundamental to our commitment to improve the overall publishing process. That is why we are proactively partnering with the research community to save editors’, reviewers’ and authors’ time, ensuring the right person is invited to review each paper, that the reviewer is properly recognized for their work, and that they are equipped to do the job in a way which improves the manuscript and advances discovery. Ultimately, the future of peer review has to be about improving our processes and using technology in a clever way so that our people can do great things.
Seth: Anonymity, many would argue, protects both author and reviewer, and allows for evaluative processes that focus on the scholarship rather than other factors. But maintaining blindness has always been a challenge in smaller fields. Now, when a few key search terms are often all it takes to reveal who is working on particular topics, it is more difficult to preserve anonymity on both sides of the exchange. While technology has made some established modes of review potentially less equitable, it has also opened up possibilities for exploring new methodologies. This, and a growing commitment to openness among many scholars, has led to experimentation with peer review that takes place with varying degrees of transparency.
Phil and Stuart: Although there can be deficiencies in its practical implementation, the principle of ‘review by peers’ has never been more important. The review of a piece of research can be almost as valuable as the research itself. Publishing peer review reports and the names of reviewers, which we and other publishers are doing, improves the transparency and accountability of the process. In the future journal peer review might be used as evidence for policy makers, as a communication tool to the public and as valid research output in awarding grants and tenure. Let’s hope so.
