Editor’s Note:  This third installment of posts for Peer Review Week is a guest post from Tom Culley and the team at Publons.

For centuries academic journals have brought modern research from around the globe into regularly published pages for consumption. At the heart of this system is peer review — the process we rely on to ensure the quality and integrity of scholarly communication. But as the research market grows exponentially the peer review system is feeling the strain.

peer review week logo

How do we know this? Publons Global State of Peer Review Report brings a new level of transparency to the state of peer review, revealing the numbers behind who’s doing it, how well they’re doing it, and how efficient the process really is. The timing is right, as the community comes together to celebrate the fourth installment of Peer Review Week, focusing on the theme of Diversity and Inclusion in Peer Review.

Released on September 7th, the report combines novel results of a global survey alongside data from PublonsScholarOne, and Web of Science. For the survey, Publons reached out to researchers via the Publons database of over 400,000 reviewers, and 1 million authors indexed in Web of Science. Of the more than 11,000 researchers who completed the survey, 69% were working at a university or college, 69% were men, and over 35% had 15 years or more experience writing and reviewing scholarly articles. The majority of reviewers came from Europe (37% — including the UK) and 13% worked in the areas of Clinical Medicine or Engineering respectively.*

The combination of these survey data — the largest response to a peer review survey — with the unique suite of cross-publisher peer review and publication data from Publons, Web of Science and ScholarOne, allowed for a truly broad and deep exploration of fundamental aspects of scholarly peer review. So what did the data show?

The report shows that the growth of the last decade — where global scientific research has doubled from 1.5 million papers in 2007 to nearly 3 million papers in 2017 — has increased the burden on editors to find able and available reviewers. During 2017, more than half of all invitations to peer review were declined. The researchers surveyed for the report state that this is because they: a) already have a large workload (42%), or b) believe the article sent to them was outside of their area of expertise (70%). It’s no wonder that 75% of academic journal editors say that finding reviewers and getting them to accept review invitations is the hardest part of their job.

Journal editors have little hope of sourcing reviewers if their reviewer pools are an assortment of fatigued and frustrated researchers. But who are they asking to peer review anyway? It’s alarming how little we actually know about the researchers moving human knowledge forward. And beyond that, it’s unsustainable to think we can continue stabbing in the dark when it comes to peer review workloads, availability, and expertise.

Diversity and inclusion: Who do we invite to review?

Editors play a critical role in the peer review process, particularly when it comes to selecting reviewers and sourcing reviews. So where are all the editors, and who are they inviting to peer review? The report shows that journal editors tend to be located in established regions, such as the United States and United Kingdom. Interestingly, editors tend to invite reviewers from their own geographical regions much more than chance would predict, implying a possible bias toward selecting reviewers from established regions.

Differences among research areas

Reviewing practices across different fields were investigated by categorizing article submission and peer review completion data. Using this approach we showed that, on average, 2.7 reviews are completed for each submission across all research areas. In tune with the high percentage of reviewers in our survey, Clinical Medicine is clearly the largest research area by publication output. On a review-per-submission basis, however, this field produces the fourth lowest number of reviews per submission.

The report also shows how some research areas have managed to speed up review turnaround times, while others remain stagnant. The median review turnaround time in Engineering, Geosciences, Agricultural Sciences, and Computer Science all steadily decreased over the five years from 2013 to 2017. Median review turnaround times in the Social Sciences, Neuroscience & Behavior, and Molecular Biology & Genetics, all remained steady between 2013 and 2017. Interestingly, the median time to complete a review in Mathematics — already the second slowest reviewing field — increased slightly between 2013 and 2017.

Regional differences in peer review: the real story

As publishers push to improve publication times, the next frontier may well be more responsive reviewers sourced from emerging regions. The phenomenal increase in published research is driven in part by the rise of China, whose article submission volumes now rival that of the US. Despite this, Chinese researchers are often overlooked when it comes to peer review.

Researchers in China are reviewing substantially less (8.8%) than the country’s article output (13.8%). Researchers in the US, on the other hand, dominate absolute contributions to peer review, submitting 32.9% of all reviews compared to 25.4% of published article output. When it comes to reviewer agreement rates and turnaround times, however, a different story emerges. Reviewers in China and other emerging regions are more likely to accept a review invitation and complete the review assignment than reviewers from established regions like the US and the UK — perhaps a sign of reviewer fatigue setting in among reviewers in these established regions.

To hone in on the theme of diversity and inclusion, perhaps — as recently suggested by the Chefs— journal editors need to start casting their reviewer net wider. Currently, editors are disproportionately selecting reviewers from (their own) established regions. The Publons Report shows that a multitude of factors account for this, including personal connections, English as a first language, and the editors’ and reviewers’ own preferences. Whatever the reasons, it is clear that reviewers from emerging regions are not being fully included in the peer review process.

Better selection methods for choosing an international mix of peer reviewers has significant potential to improve the system. The reviewing workload could be more evenly distributed and researchers from emerging regions would get greater exposure to global research.

Beyond diversity and inclusion

While the theme for Peer Review Week 2018 is an important one, it does not address all the current challenges with peer review. The scholarly community also lacks a robust measure of review quality, and this needs to be rectified fast if we want to be able to trust the next generation of published research. Proper recognition, training, and technology — and greater consensus of what this involves — are all also needed to propel the system forward.

So, once again, what does the future hold for peer review? The Publons Report puts forward some clear, immediate action points for journals, editors, institutions, and researchers to help scale to the next million articles. Beyond that, however, the system requires continued transparency of each moving part. As an industry we need to reveal the information necessary to bring a level of trust and efficiency to the peer review process. This report is a starting point, but it’s by no means the end point.

Discussion

5 Thoughts on "Guest Post: What a new Publons Report on Peer Review Says About Diversity, and More"

I have been thinking a lot about geographical diversity and the western world as the judges/gatekeepers of peer review. A frequent trope I hear from Editors that the research coming out of countries such as China, India and Iran (for example) is behind in comparison to the Western world and I think this leads them to assume the peer review they would perform would not be up to their standards. From the report I think I remember that reviews from developing countries tend to be shorter but I haven’t found any research on whether the quality of peer review from people from underrepresented countries is below standard, although this is probably due, as this article points out, that there are no defined/agreed measure of review quality. If anyone does know of any studies on geographical peer review quality I would be interested. I do think there is a geographical bias that does exist in peer review and it needs to be addressed/spoken about more. Therefore it would be great to have data on this issue.

“Better selection methods for choosing an international mix of peer reviewers has significant potential to improve the system” — I think what makes the system improve is focus on the competency of the reviewers and perhaps not the “inter-nationality” of them.

“As an industry we need to reveal the information necessary to bring a level of trust and efficiency to the peer review process.” — Absolutely. And components like ‘trust’ and ‘efficiency’ are a result of editors relying on their personal knowledge and experience to contact reviewers whom they can attest to. Precisely to preserve trust and efficiency. And that probably explains, why editors choose reviewers either from the same geographical region, or their background or some other personalized factor. This is not to say that this is the ideal way. I guess when credibility is on the line, editors tend to play safe?

Editors are under enormous pressure to meet the publishing schedule. A paper arrives and the EIC assigns it to a member of the editorial board (Sally) who has expertise in the area. Sally is running a lab, trying to finish a paper, applying for a grant, mentoring a graduate student, assisting a post doc in getting a permanent position, teaching a class, holding office hours, and trying to remember to pick up a quart of milk and loaf of bread. Now she has to get a reviewer. Her dilemma is does she go to someone she knows and probably will say yes or stop everything to seek someone she does not know or know of? On the other hand, the EIC faces paper flow problems or suffers the fate of reviewers either declining to review or just not doing the review or being late with the review they agreed to do within the time frame agreed upon. To ask editors, who have a job aside from being an editor, to begin to research acceptable reviewers from outside their geographic/knowledge area would in all probability delay publication which would anger: 1) the author, 2) the librarian who demands periodicity; and 3) the publisher who demands that the editor meets a schedule.
If the goal of reviews is to assure the paper is trustworthy, then of what difference is the nationality of the reviewer?
If there is a goal to make reviews more international in scope, why not – if not already done – put that into the role and scope statement. Then if someone from say China desires to review they can let it be known to the editor.

“The majority of reviewers came from Europe (37% — including the UK) and 13% worked in the areas of Clinical Medicine or Engineering respectively.*”
I’m sorry for being picky, but 37% is hardly a majority, there is one percentage missing (otherwise what does “respectively” refer to?), and there is an asterisk, which, from what I see, leads to nowhere.
Also, whatever the UK’s political opinion on the subject, geographically speaking it is part of Europe, like Iceland and Ireland, so I still struggle to comprehend why it needs to be stressed or resolved each time it occurs…I mean, nobody ever writes “America (including US)”, or “Europe (including Russia and Turkey)”, even if it is this last one that actually would make sense, to resolve any ambiguity.
Normally I would pass by each of these issues, but a congestion of them in one sentence just made me write this comment.

Hi Anna,

Thanks for picking up on these. It seems in my mad rush to write the article, my editing went wildly astray. Thankfully you have helped to “peer review” the piece — again showing the value of the process in improving the literature! Now to your points:

– 13% of survey respondents identified as working in the field of Clinical Medicine, and 13% identified as working in the field of Engineering. (total of 26%). Sorry, the sentence in the article is poorly written.

– The asterisk was meant to link to a note, saying that the full survey demographics can be gleaned from the survey report on this page: https://publons.com/community/gspr

– You are correct, 37% is not a majority. The correct phrase would be something like “Europe was the region with largest number survey respondents (37%).”

– As for calling out UK as part of Europe, we don’t have an opinion on it. We just wanted to make it crystal clear for readers that, in the survey, UK is included as part of Europe. We had questions on this when we first tested the survey, so we thought it best to spell it out and avoid confusion.

Leave a Comment