Note: In recognition of the role peer review has on scholarly communication, The Scholarly Kitchen will feature posts this week that celebrate Peer Review Week 2016. You can find other activities by following #PeerRevWk16 on Twitter or checking out the event listings at the Peer Review Week Website.
Peer Review Week 2016 has officially launched with a focus on recognition for peer reviewers. I thought it would be a good time to take a view of the landscape of recognition.
The two hot topics when it comes to reviewers of peer reviewed scholarship are recognition and credit. These are two very different concepts. Credit implies that a benefit is being given. We talk about authorship credit whereby an author can point to a tangible benefit of being an author, or contributor, to a publication.
Recognition may lead to a benefit but is passive when it comes to benefits. The importance of recognition is that others see the recognition and any benefit is sort of happenstance.
Publishers have no influence for giving “credit” for reviews. This is entirely up to the crediting body — the reviewers’ institutions or funders. Journals could offer certificates or websites documenting the peer review contributions of an individual but the tangible credit or reward must come from someone capable of delivering it. Several weeks ago, I went on a hunt for any institution that “credits” employees for participating in peer review of either scholarship or grant applications. I didn’t find any.
A Wiley survey of over 3,000 reviewers found that “reviewers strongly believe that reviewing is inadequately acknowledged at present and should carry more weight in their institutions’ evaluation process.”
So here we are with a portion of the academic community asking for credit and the publishers or third party startups looking to provide credit despite the fact that the who might value the credit don’t seem to have an interest. If I am wrong and there are institutions or funders in serious discussions about giving researchers professional credit for performing peer review, do let me know in the comments. I would love to hear about it.
Whether an institution wants to credit people for peer review activities is sort of arbitrary from the publisher perspective at this point. The publishers are most definitely looking for ways to recognize peer reviewer contributions, and if that recognition leads to reviewers getting some sort of professional credit, that would be fantastic!
Since we aren’t the parties most capable of offering career advancement rewards, instead we need to ask ourselves, what can we offer that makes a difference? What do peer reviewers want most from journals? What keeps them coming back as reviewers? What kind of recognition motivates them most? We have several reports and surveys with some of these answers:
- Feedback: When asked what makes the reviewer most likely to review with a journal again, the top two answers in the Wiley survey were feedback from the journal on the usefulness or quality of the review and notification on the final decision made by the editor.
- Acknowledgment: Being named in a printed list of reviewers, reviewer of the year awards, and a certificates acknowledging review were also highly valued by respondents of the Wiley survey. Likewise, a Sense about Science survey found that 40% of respondents valued being publicly acknowledged by the journals.
- Payment in cash: There is a lot of controversy over paying reviewers. The “pro” argument states the value of the reviewer’s time, the fact that cash is almost always a good motivator, and that some publishers make a lot of money. The “cons” argument questions the conflicts inherent in a paid reviewer system. The Sense about Science survey found that 41% respondents favored cash payments for reviewing but that number dropped to 2.5% when told that the payment would be covered by the author. In contrast, the Wiley survey found that payments to peer reviewers were not a strong incentive for review.
- Payment in kind: The Wiley survey, the Sense about Science Survey, and a survey by Taylor & Francis all found that reviewers were favorable to receiving payments in kind such as discounts on Open Access (OA) fees, access to content, free color in their papers, and continuing education credits.
The payment option in cash or in kind carries administrative issues for publishers that really cannot be discounted. Reviewers have also stated in comments on blogs and on social media that the hassle of receiving payments that accumulate to an amount that requires tax forms to be filed is ridiculously counterproductive. Providing access to content is only valuable if the reviewer doesn’t already have access to that content, say through their university libraries. Discounts on OA fees or color charges seems more probable.
Moving on to the top rated incentives of feedback and acknowledgement, there is much that can be done. If you spend a few hours writing a review, you want to know if it is helpful. At a recent editorial board meeting for one of the ASCE journals, we talked about ways to make it easier for editors to personalize a thank you to reviewers and add comments about the review.
Journals should also send each reviewer the decision letters and associated comments to the authors. This is an important part of training. It also allows the reviewers to see the other reviews and try to analyze any differences.
Many journals also publish a list of reviewers each year. Going a step further, several organizations, including my own, have a program for recognizing Outstanding Reviewers. Editors select their best reviewers who then receive a certificate and a letter recognizing their contribution. We send the letter not only to the reviewer, but also their supervisor. A separate list of outstanding reviewers is published online. You can see examples of this here and here.
A somewhat controversial version of publicly acknowledging top reviewers was announced recently for the Journal of Behavioral and Experimental Economics. This pilot program will rate reviewers and then publish their names with their ratings. Those who rate really low will be spared the public shaming and, presumably, forego any public recognition for the contributions they may have made. The controversial part of this seems to be centered on the criteria for ranking, which is based entirely on speed of review.
Other publisher and third-party initiatives are aimed at provide a public glimpse into reviewer output.
Publons: Chances are you’ve been hearing about Publons for a while now. Straight up, Publons is a way for users to create a reviewer profile that includes all the journals they are affiliated with. Further, publishers can validate the review activity and depending on the journal policies, the reviewer may be permitted to list the papers reviewed and publish the reviews.
My Elsevier Reviews Profile: Elsevier has a new reviewer profile platform that also acknowledges the peer review activity of individuals. There is a certificate component as well as discount codes for Elsevier services. Reviewers can pull reports of their yearly activity, which may assist reviewers with US entry Visas that have a need to prove reviewer contributions.
ORCID: Publishers can use ORCIDs collected from reviewers to report their review activities on their ORCID profiles. How the publishers or agencies will transmit this information is a little fuzzy but “early adopters” include some of the major manuscript submission and review systems as well as Publons. The goal is to include all peer review activities, not just publications related.
Frontiers: This publisher has been publishing reviewer names alongside published papers but has now added reviewer activity to the Loop page (user profiles) on the platform.
Open Peer Review: There are journals that are publishing reviews and reviewer names alongside the published papers. F1000 Research, select BioMed Central journals, and The EMBO Journal (among others) are publishing reviewer names, which could be the ultimate form of recognition if this process works for the given community.
There is a flaw to some of these reviewer recognition initiatives. Papers under review are treated as confidential and as such, reviewers on a rejected paper will not qualify for any recognition tied to the publication of a paper. Outstanding reviewer programs and simple numbers (this person reviewed 5 papers for us) will provide recognition but will not be as granular and perhaps not seen as valuable. In establishing recognition programs, we must not forget the enormous contribution of the reviewers of declined papers.
As we celebrate Peer Review Week 2016, I am very interested in hearing from publishers what they are doing to recognize and incentivize reviewers; from institutions and funders whether they are moving toward credit for reviewers; and from reviewers what they want to get out of the system. I look forward to your ideas in the comments!
13 Thoughts on "Is More Recognition the Key to Peer Review Success?"
Hi Angela, thanks for this great post. We’re delighted to be involved with Peer Review Week! At IOP we’re going to be trialling a number of initatives in the next few months, most of which will be announced this week. Our first announcement is that we’ll be doing Reviewer Awards from 2017, acknowledging our top reviewers for the previous year according to quality, quantity and timeliness. You can keep up to date with our other announcements at http://iopscience.iop.org/page/peer-review-week, which also includes videos, infographics, tips for reviewers and details of our Peer Review twitter chat on Friday this week.
The publishers can make semantic search and analysis available to reviewers giving them the same advantages as researchers outside of academia. The rationale is straight forward. It saves time, catches errors and duplications and, when also presented to authors gives them a chance to comment, revise or even withdraw without committing time and possible exposure to academic reviewers. It also provides the foundational analysis for critical reviews and increases the value of such reviews
I’m the old curmudgeon. Peer review is a contribution not a paid activity nor one that gets one a pat on the back. If one wants a pat on the back for contributing to one’s profession, go to the tenure committee and request one. I am sure the committee will see fit to promote you because you carried out the most reviews, of the highest quality, and in a timely fashion!
In short, if you don’t want to do a review don’t!
The American Physical Society has had a ‘Outstanding Referees Program’ since 2008.
People should be reminded that this post covers only peer reviewing for journal articles, not for monographs, where for example the payment of cash or in-kind (in the form of free books) honoraria is the common practice. What both have in common is that they fall under the rubric of “service” activities, which in the academic triad of research, teaching, and service has traditionally been the least valued of the three in tenure and promotion decisions. That is not likely to change anytime soon, but efforts such as those highlighted in this post could go some way toward enriching the status of this kind of service activity.
Nice post Angela. Publons can, and does, track and verify reviews for rejected manuscripts. We think it’s really important that every review done is recognised as a valuable contribution. That way individuals and institutions can start showing how much influence they have in their respective field, or on research more generally, through their peer reviewing activity.
I also want to point out that some institutions and funding bodies are wising up to the importance of peer review and factor peer review contributions into promotion and funding decisions. For example, the PBRF in New Zealand and ARC in Australia both ask for data on non-traditional research outputs for funding rounds. Peer review data is precisely the type of thing that can help set an application apart and secure funding.
Similarly, institutions are factoring peer review contributions into promotion and tenure decisions, as well as awards considerations. Dr Matthias Lein is a case-in-point of an individual researcher using proof of his reviewing prowess to help get a promotion (https://publons.com/static/Publons%20-%20Matthias%20Lein%20promotion%20case%20study.pdf).
It’s true these funding and career progression benefits aren’t exclusively a result of one’s reviewing expertise, but review contributions do factor in. I do think there is more to be done on this front, but it’s a start. At Publons, we’re working hard to raise the status of peer review and turn it into a standard and measurable research-related output.
The most extensive recognition of reviewers I’ve experienced is at the American Educational Research Association, where each journal designates 10 outstanding reviewers each year and invites them and their deans to a reception at the AERA Annual Meeting. Each reviewer gets presented a certificate, and of course there is a public news story on the website. This public celebration witnessed by their deans is a recognition that really equates to a tangible career credit.