Editor’s Note: This third installment of posts for Peer Review Week is a guest post from Tom Culley and the team at Publons.
For centuries academic journals have brought modern research from around the globe into regularly published pages for consumption. At the heart of this system is peer review — the process we rely on to ensure the quality and integrity of scholarly communication. But as the research market grows exponentially the peer review system is feeling the strain.
How do we know this? Publons Global State of Peer Review Report brings a new level of transparency to the state of peer review, revealing the numbers behind who’s doing it, how well they’re doing it, and how efficient the process really is. The timing is right, as the community comes together to celebrate the fourth installment of Peer Review Week, focusing on the theme of Diversity and Inclusion in Peer Review.
Released on September 7th, the report combines novel results of a global survey alongside data from Publons, ScholarOne, and Web of Science. For the survey, Publons reached out to researchers via the Publons database of over 400,000 reviewers, and 1 million authors indexed in Web of Science. Of the more than 11,000 researchers who completed the survey, 69% were working at a university or college, 69% were men, and over 35% had 15 years or more experience writing and reviewing scholarly articles. The majority of reviewers came from Europe (37% — including the UK) and 13% worked in the areas of Clinical Medicine or Engineering respectively.*
The combination of these survey data — the largest response to a peer review survey — with the unique suite of cross-publisher peer review and publication data from Publons, Web of Science and ScholarOne, allowed for a truly broad and deep exploration of fundamental aspects of scholarly peer review. So what did the data show?
The report shows that the growth of the last decade — where global scientific research has doubled from 1.5 million papers in 2007 to nearly 3 million papers in 2017 — has increased the burden on editors to find able and available reviewers. During 2017, more than half of all invitations to peer review were declined. The researchers surveyed for the report state that this is because they: a) already have a large workload (42%), or b) believe the article sent to them was outside of their area of expertise (70%). It’s no wonder that 75% of academic journal editors say that finding reviewers and getting them to accept review invitations is the hardest part of their job.
Journal editors have little hope of sourcing reviewers if their reviewer pools are an assortment of fatigued and frustrated researchers. But who are they asking to peer review anyway? It’s alarming how little we actually know about the researchers moving human knowledge forward. And beyond that, it’s unsustainable to think we can continue stabbing in the dark when it comes to peer review workloads, availability, and expertise.
Diversity and inclusion: Who do we invite to review?
Editors play a critical role in the peer review process, particularly when it comes to selecting reviewers and sourcing reviews. So where are all the editors, and who are they inviting to peer review? The report shows that journal editors tend to be located in established regions, such as the United States and United Kingdom. Interestingly, editors tend to invite reviewers from their own geographical regions much more than chance would predict, implying a possible bias toward selecting reviewers from established regions.
Differences among research areas
Reviewing practices across different fields were investigated by categorizing article submission and peer review completion data. Using this approach we showed that, on average, 2.7 reviews are completed for each submission across all research areas. In tune with the high percentage of reviewers in our survey, Clinical Medicine is clearly the largest research area by publication output. On a review-per-submission basis, however, this field produces the fourth lowest number of reviews per submission.
The report also shows how some research areas have managed to speed up review turnaround times, while others remain stagnant. The median review turnaround time in Engineering, Geosciences, Agricultural Sciences, and Computer Science all steadily decreased over the five years from 2013 to 2017. Median review turnaround times in the Social Sciences, Neuroscience & Behavior, and Molecular Biology & Genetics, all remained steady between 2013 and 2017. Interestingly, the median time to complete a review in Mathematics — already the second slowest reviewing field — increased slightly between 2013 and 2017.
Regional differences in peer review: the real story
As publishers push to improve publication times, the next frontier may well be more responsive reviewers sourced from emerging regions. The phenomenal increase in published research is driven in part by the rise of China, whose article submission volumes now rival that of the US. Despite this, Chinese researchers are often overlooked when it comes to peer review.
Researchers in China are reviewing substantially less (8.8%) than the country’s article output (13.8%). Researchers in the US, on the other hand, dominate absolute contributions to peer review, submitting 32.9% of all reviews compared to 25.4% of published article output. When it comes to reviewer agreement rates and turnaround times, however, a different story emerges. Reviewers in China and other emerging regions are more likely to accept a review invitation and complete the review assignment than reviewers from established regions like the US and the UK — perhaps a sign of reviewer fatigue setting in among reviewers in these established regions.
To hone in on the theme of diversity and inclusion, perhaps — as recently suggested by the Chefs— journal editors need to start casting their reviewer net wider. Currently, editors are disproportionately selecting reviewers from (their own) established regions. The Publons Report shows that a multitude of factors account for this, including personal connections, English as a first language, and the editors’ and reviewers’ own preferences. Whatever the reasons, it is clear that reviewers from emerging regions are not being fully included in the peer review process.
Better selection methods for choosing an international mix of peer reviewers has significant potential to improve the system. The reviewing workload could be more evenly distributed and researchers from emerging regions would get greater exposure to global research.
Beyond diversity and inclusion
While the theme for Peer Review Week 2018 is an important one, it does not address all the current challenges with peer review. The scholarly community also lacks a robust measure of review quality, and this needs to be rectified fast if we want to be able to trust the next generation of published research. Proper recognition, training, and technology — and greater consensus of what this involves — are all also needed to propel the system forward.
So, once again, what does the future hold for peer review? The Publons Report puts forward some clear, immediate action points for journals, editors, institutions, and researchers to help scale to the next million articles. Beyond that, however, the system requires continued transparency of each moving part. As an industry we need to reveal the information necessary to bring a level of trust and efficiency to the peer review process. This report is a starting point, but it’s by no means the end point.