Editor’s Note: Welcome to the dog days of summer. So many of the pressing issues in scholarly communications are not new, but rather continuations that have evolved over years if not decades. As our Chefs enjoy the waning days of summer, we’re taking a look back at earlier posts on subjects and offering thoughts on what, if anything, has changed.
Last week’s post by Alice Meadows showcased some of the impressive new technologies that are better enabling us to track the work done by researchers, in particular, work that has previously remained somewhat invisible. But even with these abilities in-hand, functional questions still remain about who would grant that credit and what form it would take.
I was inspired by the comments on Alice’s post to go back and look at The Scholarly Kitchen archives to see how long the conversation over credit for peer review has been going on. The first mention I can find is in this 2009 piece by Phil Davis, looking at a survey done that year by Sense About Science. One of the key takeaways from that survey is that acknowledgement, either in the form of payment or career credit was sought after by more than half of the respondents. So although the conversation probably goes back further, we can definitively say that this has been in discussion for more than ten years now. Systems offering credit for peer review date back to at least 2010, with the proposal for PubCreds, an attempt at privatizing the peer review commons that never got off the ground.
So I’ll pose the question as we revisit my 2015 post about credit for peer review — are we any closer to realizing it?
Offering career credit to researchers for performing peer review seems like a no-brainer, right? Peer review is essential for our system of research, and study after study confirms that researchers consider it tremendously important. Funding agencies and journal publishers alike rely on researchers to provide rigorous review to aid in making decisions about who to fund and which papers to publish. On the surface it would seem to make sense to formalize this activity as a part of the career responsibilities of an academic researcher. But as one delves into the specifics of creating such a system, some major roadblocks arise.
One such problem falls into the realm of volunteerism and motivation. Right now, most academics see performing peer review as a service to the community. It’s important to the advancement of the field and so they volunteer their time. If instead we turn peer review into a mandatory career requirement that is rewarded with credit, it changes the nature of the behavior. If we set standards (you must do X peer reviews per year) people will then work to those standards rather than the more generous acts we see today, where good samaritans (and good reviewers) take on much larger workloads.
Economists suggest that incentives (a form of reward) changes motivation, some of which will be actualized by real behavioral change. Educator Alfie Kohn talks about how behaviors change in light of offering rewards in one of his books on parenting:
…there are actually different kinds of motivation. Most psychologists distinguish between the intrinsic kind and the extrinsic kind. Intrinsic motivation basically means you like what you’re doing for its own sake, whereas extrinsic motivation means you do something as a means to an end — in order to get a reward or avoid a punishment…extrinsic motivation is likely to erode intrinsic motivation…The more that people are rewarded for doing something, the more likely they are to lose interest in whatever they had to do to get the reward.
Plugging peer review into a rote system of requirements may threaten both the participation levels and the rigor and enthusiasm with which many approach the task.
A second problem with peer review credit schemes (and perhaps I’m arguing against my own interests here) is the increased power it places in the hands of publishers and editors. Researchers have little control over whether they are asked to do peer reviews. If your career is dependent upon getting a journal to ask you to review a paper, what happens if you’re not on their list? An early career researcher who is not well-known in their field is at an automatic career disadvantage under such a scheme. Many are already resentful of the “kingmaking” ability of editors of journals like Science, Nature and Cell. Turning over even more power over academic career success to publishers might not be so well-received.
Perhaps the biggest problem of all comes when we ask the simple question that must always be asked when new changes to the academic career structure or the scholarly publishing ecosystem are proposed: Who cares? Not “who cares” as in “peer review is unimportant and no one should care about it”, but “who cares” as in “who exactly are we asking to grant credit here?”
As we are constantly reminded, the two things that matter most to academic researchers are career advancement and funding (and the more cynical among us suspect that the former is primarily dependent on one’s ability to secure the latter).
If I was an administrator at a research institution, I’m not sure I’d want my researchers spending an enormous amount of their time helping to improve the papers of researchers at other institutions. If I was a particlarly wise administrator able to see the big picture, I would understand the value of peer review and how it is necessary for the advancement of knowledge. So I’d know some amount of credit is due. But it’s not the primary reason I hired those researchers, nor is it something I want them spending a lot of their time doing. Their job is to do original research.
Similarly, a funding agency gives a researcher a grant to do research. A diabetes foundation is looking to fund research to cure the disease and likely wants fundees spending their time doing original research, not reviewing papers from other researchers. How much should they reward fundees for doing something other than what they’ve been funded to do? And back at that research institution, if much of the tenure decision (at least in the sciences) is based on how much funding one can bring in, then if peer review doesn’t bring in funding, it won’t matter all that much.
How much career credit should a researcher really expect to get for performing peer review? I suspect that at best it will be a few small percent of the overall picture, more likely a box on a checklist–did you do any peer review? If yes, then you get a small bonus amount of credit. No one is going to get hired, tenure or funding based on a stellar record of peer reviewing lots and lots of papers. There’s a different job where you get rewarded for that — it’s called “editor”.
The proposed peer review credit systems currently under examination, both commercial and community-based seem like overkill. Many systems offer extensive tracking, point systems and review of reviewers which may be unnecessarily complex for a yes/no question. As Joe Esposito has trained us to ask, this is at best a feature, certainly not a product nor a business.
Perhaps something along the lines of the work ORCID and CASRAI are doing will suffice in the end. Tag the activity to the researcher’s identifier and offer a simple yes/no or a tally of peer review events for the year. Do we really need anything more than that?
Discussion
28 Thoughts on "Revisiting: The Problem(s) With Credit for Peer Review"
Dear David,
Thanks for your nice overview. To answer your final (rhetorical) question: Yes we need more than that. I agree that the existing and proposed reward systems all have their disadvantages, but I don’t think that leaving things as they are is a tenable solution, as the system is reaching its boundaries (or in fact already has crossed them). Too much time in our global process of knowledge production is lost, because a system based on charity is kept that delays the whole endeavor. While writing a review takes half to one day, time to first decision in social sciences, humanities and other fields is three to four months (on average, so often much longer). And over half of papers is rejected, so after waiting so long the the author can in the majority of cases start all over again. Non of the mentioned reward systems is able to solve this issue. The only solution is real professionalization, which means paying reviewers for their work. Only with payment you can demand reviewers to deliver their reports quickly (say in 10 days). As anybody else in the system is paid for their work, why not the reviewer?
Where paying peer reviewers has been tried (e.g., Company of Biologists), it did not see much success — in their case, they stopped the practice at the request of reviewers who felt that the efforts of filling out the forms necessary to be paid by the journal were not worth the small amounts offered. And that’s one of the central problems — how much would you pay a peer reviewer to make it worth their time? What would you have to pay the Dean or Department Chair at a major research institution to convince them to turn around a review in 10 days? Unless it’s a significant amount, it’s only going to be attractive to very junior people, who may not be the ideal, experienced, and knowledgable peer reviewers you’re seeking.
That money would also have to come from somewhere, so that means increased subscription prices or higher APCs. I don’t see academic libraries, researchers, or funders volunteering to spend lots more money on publication costs at this time. One could argue that the large commercial conglomerate publishers could take it out of their profit margins, but this is 1) unlikely given their business practices, and 2) not at all feasible for smaller or not-for-profit publishers. Even if you could get buy-in from the corporations, it would essentially drive everyone else out of business. That small Diamond OA journal published out of a university library is not going to be able shell out large sums of cash for every article submitted.
From another angle, as noted in the post above, making review a paid activity rather than a community service activity changes its nature. Most journals I know have a pool of strong reviewers that they rely on fairly heavily, and those reviewers often see their work as a service to their community/research society. If instead it’s a paid transaction, they will likely view it differently and instead put their time toward more monetarily rewarding activities.
Jeroen: Just how much is a review worth monetarily? Is one going to drop everything for say $50 or $100 or $1,000? Who is going to pay for the review? The publisher will just tack on the cost to either the subscription or the author’s fee. But, someone is going to pay. There is no free lunch even for reviewing! It seems under the current system one reviews because one wants to and their motivations are really unknown!
Reviewing for acknowledgement is an interesting thought. I can see it now: Joe is sitting and having coffee in the faculty club and says: Did you see my latest review in the Journal of Nonsense. It is my 20th this year. I am sure (he says to himself look they are jealous!) that all will notice it!
It seems to me that Donald Forsdyke expresses what makes scholarly pursuit different and at the same time common among those in academia.
Spot on David! When invited to engage in pre-formal-publication peer review, I accept if the topic is within my field of interest, partly because I feel duty-bound, but mostly because it might help both me (I might learn something) and fellow researchers (the authors and others who may eventually read the paper). And even when not-invited, I review (e.g. by way of PubPeer) both preprint and formal publications to help fellow researchers and readers. Why? Because research is darned difficult and I may have spotted something that neither authors nor the editor-chosen reviewers had considered. In this respect, I continue to mourn the loss of PubMed Commons.
Thanks David. You use the word “reward” multiple times in this posting. It suggests simplistic and transactional motivations.
What if you used the word “acknowledge” or “recognize” instead?
Peer review recognition is only one example of the need for improved research contribution recognition. Researchers (especially early career researchers) go unacknowledged for many contributions that are not usefully recognized in our citation-centric research culture.
Citations are a great way to show how ideas are connected but they are a useless, counterproductive and antiquated tool for recognizing individual contributions. It’s about time we dump citations as the fiat currency of research contribution recognition.
Richard Wynne – Rescognito
Not wanting to get too deep into semantics, what value is offered the researcher by that “acknowledgment” or “recognition”? Who does it matter to, and what will they do with it?
That some reviewers hand in their reports in a short time does not mean that the system works. The long average duration of the process proves that it does not work.
Most positive examples are of high level journals or of the medical sector (where papers are short and much money is available), but there are over 10,000 middle class journals that increasingly have problems getting their reviews back. It is in these journals that by far most researchers publish their work (by definition). I would like to hear your solutions for bringing back review time for these journals to a reasonable two weeks (for a report of one day work), without introducing a paid system.
Regarding the involved costs, you may remember that 20 years ago nobody could believe that a researcher/university would be prepared to pay between 1000 and 5000 dollar for open access. And now this system has become mainstream.
The price could be something like 100$ per review. There are many researchers in the world who would be willing to deliver a good quality review within a week for that price. Only 20% of the world population lives in a wealthy country. Remember that we are talking about the large mass of middle class journals and not the highest levels. So we don’t need the dean or department chair as reviewer. A regular academic in India or China might do the job just as good.
So for every paper, costs would go up by $300 for three peer reviews. But wait, that’s not accurate, as not all papers that are reviewed are accepted. Let’s posit a journal with a 50% desk rejection rate, and an overall 70% rejection rate. For every 100 papers that come in, 50 are rejected immediately, 50 are sent out for peer review, and 30 are accepted. To get 30 articles accepted, 50 are reviewed, so the cost is $15K, but that cost is only borne by the accepted papers, so it works out to $500 per article. Then if any papers are asked for revisions and re-reviews, that’s another $300 per paper. Then you have to pay for someone to monitor and issue all of those payments and collect and maintain tax information from every single reviewer, you have to pay for an e-commerce system to deliver the payments, you have to do customer service to deal with reviewers whose payments have gone astray, etc.
And you’re assuming that the APC-author-pays model of OA is the only business model that needs to deal with these payments. It would rule out all community-run journals that are free to authors/readers, it would make business models like Subscribe to Open and community supported models much more expensive, it would significantly increase subscription fees, and it assumes that the world is okay with paying $5K per article (it’s not) and wouldn’t balk at tacking on another $1K or so.
It would shut out authors from LMIC countries who already can’t afford APCs at current levels, and it would grant a huge competitive advantage to big commercial publishers — why peer review for your society journal that only pays $100 when you could instead review for an Elsevier journal that pays $150?
I don’t have an answer to your question as to how to get reviews back faster, but like the APC model for OA, paying reviewers creates as many problems as it solves.
So in round numbers: $100 per review, 3 reviews per *submitted* paper, 4 million submitted papers a year. That’s an increase in cost structure of $1.2 billion a year. We can argue about the numbers, but you get the drift. At that cost many publishers would cut back or drop peer review entirely.
$100 per review? So if the average review takes me between half and a full [working] day to (as you say above), then that’s 4-8 hours. Let’s split the difference and call it 6 hours. In other words, you’re proposing to pay people with PhDs less than $20 an hour?
Please note the CASRAI documentation of the CreDiT taxonomy is in the process of being migrated to http://credit.niso.org/ (which should probably be an https:// site) and links to https://casrai.org/ are not working.
Hi David,
Yes, I would think that an author would have to pay 500$ upon submitting a paper for the pay review process, and if the paper is rejected again the same amount for a new submission. The reviewer will obtain 100$ for the whole process, including reviewing re-submissions, and is paid at the end of the process.
This system does not shut out authors from LMICs. In fact, a great advantage is that it would give researchers from low income countries the possibility to earn money by doing reviews (and in this way pay the reviewing fees for their own papers), which is better than having to beg for reductions.
I understand that from the perspective of publishers and editors this might complicate things, but in the end the whole system exists on behalf of the researchers and to facilitate their work. We all know that the real problem for the bulk of authors is the long duration of the review process and the uncertainty of the system. Many authors in the middle field of science know the experience of waiting nine months or more to get a rejection on the basis of one or two short review reports. For PhD researchers these experiences are terrible. The whole open access discussion and all the money spent on it has changed nothing to this great problem.
So if we are prepared to spent thousands of Euros per published paper on open access — which does really solve any problem for the authors — why not also spent money on something that is really helpful for them? I don’t think that at Oxford Press there is any person working for free. Why should researcher have to do this?
As noted above, $500 would not account for monitoring and administration of such payments, so my guess would likely be somewhat higher. Regardless, one also has to realize that it would not be a one-time payment, but instead something that gets paid over and over again as the paper may not be accepted to the first several journals to which it is submitted. I don’t agree that authors from LMICs are prepared to spend thousands of euros per paper on open access either, and earning the occasional $100 for a review (an invitation to which they may never receive) does not solve that problem. As we’ve seen from floating the idea of submission fees, these additional costs are not going to be well-received by the community.
I would also argue that having to wait a couple of extra weeks for a peer review decision is pretty far down the list as far as problems facing most researchers (start with the lack of funding and jobs and go from there).
I don’t think that at Oxford Press there is any person working for free. Why should researcher have to do this?
I perform peer review for journals all the time, as well as writing the occasional article or editorial. I am not paid any more than my usual salary when I take on these tasks, nor would I be docked any salary if I declined the invitation. Along with my employer, I see these activities as part of my already salaried job, contributing to the betterment of the community. I suspect that most researchers view peer review in the same light.
Jeroen: Reviews always took time. Today, they are faster than in the past. You forget or never experienced publishing before computers! In those days of yore one awaited the mail! International mail mostly went by ship and domestic by train! Talk about delay….
So you are suggesting that scholars in ‘third world’ countries are willing to work for less than those in “first world” countries! Scholarship is complex and the ability to convey ideas requires strong language skills. A review is worthless unless clearly presented. Have you read unedited manuscript from authors from “third world” countries? So onto the cost of the review one would have to add the cost of editing the review not to mention time! Few problems have easy answers and non-existent ones are even more complex!
Hi Harvey,
I do not agree that reviews are currently quicker than in the past. In the 1990s we submitted large envelopes with three one-sided double-spaced paper copies by regular mail. That took some time, but if you received one of those copies from a journal with the request to review it (or otherwise send it back to the journal) you hardly ever refused. Nowadays you can refuse with one button press and many researcher do so, simply because they are overloaded with work. That 30 years later first response time is still on average three to four months in many scientific fields is the best evidence that the system is failing.
Regarding your remark about reviews and manuscripts from third world countries, I do not believe that there are no good scholars in LMICs (where 80% of the world population lives). Moreover, in a paid peer review system, reviewers can be given a quality score by the editor and author(s) and reviewers with better scores get more papers to review. In this way, rather quickly a pool of professional reviewers might come into existence. And I am sure that many of them will be found in a country like India.
#Joe: If these are the costs needed to pay researchers for their work, it should be paid. How big are they compared to the costs of open access, which does not solve the real problem of researchers at all? Nobody in the system works for free. Why should researchers do so?
#Shaun: In most part of the world this is a reasonable price for a day work of a professional. It would also make more research money move to LMICs, which I don’t believe is bad.
I think you should start the company you envision. The proof is in the execution.
Hi Joe,
The solution is not a new company. There are already too many grazing on research money. What we need is a real discussion on how the peer review process can be brought back to two weeks. I believe that this can only be achieved by paying reviewers for their work. I invite you to come up with better ideas instead of only criticizing. How can we solve this problem?
I’m not sure I quite grasp why a peer review process that takes a measured pace is the biggest issue facing researchers. Karin Wulf wrote a great piece in 2017 about embracing the slow: https://scholarlykitchen.sspnet.org/2017/08/09/pace-intellectual-publishing-embrace-slow/
But if this is the case, then solutions that don’t cost billions of dollars out of research budgets:
1) Mandate preprints — this puts the work out for public consumption and comment while the peer review process is happening
2) Create incentives to reduce the number of papers published — the new policies out of China do just this, only counting a researcher’s best papers rather than counting the quantity of papers produced. This would reduce the volume of peer review requests on the community, offering better efficiency.
3) Set time limits for editorial processes — if a journal can’t find enough reviewers to process a paper in a given time, then return it to the author to submit it elsewhere.
I beg to differ: You are the one who is only criticizing. You have a hypothesis. That’s great. Now test it. That’s what new companies are for. If you aren’t going to start a new company, why bother with the critique? In any event, a new company would lessen the demands on research funds by introducing new industrial processes. The tried and true way for new companies to add value is to reduce costs.
#Joe: I don’t have a hypothesis, I have a problem. As a researcher I face a system on which I am dependent and which is not working well, as it uses over three months for work that takes only one day. This is a great problem for me and my colleagues, as our career depends to a large extent of publishing papers. If you work for a year on a specific project and you have finally written and submitted your paper, it creates much uncertainty if you have to wait for five months (which often happens) and it then comes back with a rejection and you have to start over again. This is particularly so for young scholars. The scientific field is highly competitive and the reviewer who rejects the paper might be (often is) somebody working on a related topic, or even somebody working on the same topic, so any unnecessary delay is problematic. This is one of the biggest problems science is facing and it gets too little attention. If we talk about credits for peer review, as in David’s blog, we are talking about ways to solve this problem. I believe that a good way to do this is by paying reviewers for their work, as this is the only way I can see in which you can require them to deliver quickly. If you know other ways, please let me know.
p.s. SciRev is not a company but a nonprofit organization which aims to improve the peer review process by offering researchers the possibility to share their experiences with the review process.
It happens rarely, but, occasionally, I ask a colleague to review a paper and (s)he responds, “Sorry, I don’t have the time.” I then ask the colleague to make time, and remind him that those who publish have a moral obligation to review because…
Our journal requires a minimum of two reviews before an editor can make a decision. The average number of reviews per paper is three. Even if the acceptance rate is as high as 50%, for every accepted paper six reviewers, on average, contributed a review. Therefore, for every published paper, the authors have a moral obligation to review about six papers.
When I know the refusing person well, I also, on occasion, have told him that the next time I’m asked to handle the review of one of his papers, I’ll remember that he declined to do a review when I asked him.
Do onto others…
Back to Roy Kaufman’s idea of submission fees. A pool created by submission fees could pay the reviewers.
There’s a simple thing publishers could do to improve the equability of peer review invitations, reduce editor frustration, and speed article turnaround: list contact information for ALL authors, not just the corresponding author. Many times in my search for reviewers I find a closely related paper with Professor Big Cheese listed as the senior and corresponding author, and I cannot find contact information for the first author. First authors are often early-career scientists or advanced students without a generally accessible web presence. The Professor Big Cheeses can just be too busy and too important to review or even to bother to suggest a junior colleague. And early-career scientists need to be able to list their fleeting institutional and permanent, personal emails.
A related area involves translators, significant in my field, European medieval history. More precisely, editors must correct the “English” of many papers they receive. I’ve been called upon to peer-review unedited submissions, before correction. It isn’t always fun. Looking at much published work, it’s fairly obvious the non-anglophone authors did not write in perfect English, yet there is no credit for the translator or whoever corrected the mess. This is deceptive.
Dear David and all,
I am new to the forum so would like to introduce myself before commenting on your very important piece. My name is Giacomo Bellani, I am an associate professor of Intensive care at University of Milano-Bicocca and a co-founder of ReviewerCredits (https://www.reviewercredits.com), a spin-off of the university created a couple of years ago that only in the last few months has taken off as a structured start-up.
As you can tell from its name, ReviewerCredits operates exactly at the core of the points discussed in your article. As a clinician, a researcher, an author, a reviewer and a journal editor I can only echo several of the points you and other raised, including the complexity of the problem due to the lack of sustainability of a peer reviewing process based on economic transactions.
The point at the core of ReviewerCredits proposal is the attempt to contribute to the system by creating a virtual circle by which Reviewers registered in our platform accrue credits that can then be redeemed in activities that currently are either (a) useful for the reviewer’s own academic careers (such as discount on training programs, APC fees etc) or (b) instrumental in initiatives that are helping the environment (https://www.reviewercredits.com/store/).
With this model, whilst there is no monetary transaction, there is at least a recognition that the time spent by the reviewers is somewhat “redeemable” in areas that are very coherent with the progress and sustainability of science and the environment.
Furthermore, we feel that the “certification” of the peer reviewing activity by the platform contributes alongside other very important companies such as ORCID and Publons to raise awareness of the time and effort spent by researchers and academics, ultimately contributing to “raise” the image and importance of peer reviewing as a pivotal activity going forward.
Publishers may contribute to this virtual circle by acting as a multiplier, so that reviews in registered journals earn credits for the reviewers more quickly and efficiently
We are also seeking to introduce elements looking at the timeliness and quality of the peer review, alongside the effort of the PEERE initiative, although we acknowledge that it is early days to be able to tackle these even more complex nuances of peer review.
Our aim is for ReviewerCredits to become a potential contributor of the long term sustainability of this critical activity. We would obviously be very intrigued to hear what you and other feel about such model of “virtual credits” and its scalability.
I love the idea of Continuing Education credits for peer review activity (though like CE, someone would need to verify and monitor such activity). That and discounts on society membership or training programs would be helpful incentives and give research society publishers an advantage over publishers that are not certified for CE nor able to offer member services or training. In some areas for some researchers, it would answer the question of “who cares” (the research society) and “why bother” (CE credits and discounts on things you were going to spend money on).
It’s not a universal answer though, as it doesn’t easily apply to other fields (no Continuing Education requirements for most non-medical fields for example) but offers some interesting possibilities.
Thanks a lot David. This is indeed a challenge: looking for benfits which might be “trasversal” and satify reviewers from several disciplines. This is what we try to do in our “virtual store” whare we allow reviewers to get discounts on APCs, training courses, conferences or even help the environments to plan trees
