How do you motivate reviewers? In a voluntary market, journal editors provide non-financial rewards to incentivize researchers. Many shower their best reviewers with accolades at national conferences, print their name in journals, and consider their best and brightest reviewers for prestigious positions on their editorial boards. Most academics will gladly work for free if treated well and lauded with the respect of their peers.
The trusted relationship between editor and reviewer takes time and, like many trusted relationships, cannot be taken for granted. In return for loyalty, an editor must carefully select which manuscripts — and how many — are sent to each reviewer. As many editors understand deeply, it is easy to upset reviewers by sending them inappropriate manuscripts (out of scope, poorly written, or poor quality), or by burning out the best with far too many requests to review. Like any organization that runs on voluntary labor, there is no obligation for a reviewer to stick around. Reviewing is perceived as a favor, not a job.
In last Friday’s post, I explored the notion of paying reviewers for their time. While Rubriq is proposing paying reviewers $100 to incentivize timely reviews, serious discussion of paying reviewers goes back over 20 years to an article published in 1992 in (not surprisingly) an economics journal. Publication speed in economics is believed to be the slowest in the industry, with authors waiting 2-3 years between submission and publication for some journals. With this lag, it is not surprising why working papers are part of the publication pathway for economists. In addition to paying reviewers, economists proposed blacklisting slow reviewers. One even wished for the untimely death an ineffectual editor.
Today, reviewers for journals published by the American Economic Association are payed $100 for “timely reports.” It is not clear from the journal website what is meant by “timely.”
Rather than speculate whether an independent review company with no connection to a learned society or academic association (e.g. Rubriq) would succeed on paying reviewers $100 for their services, I set up a straw poll last Friday to gather results from our readers. I’m among the first to recognize the many potential biases in these surveys (sampling bias, response bias, leading questions, priming, among others); however, the results yielded some interesting results:
- Editors are hesitant about outside peer-review. Of the 125 votes, just 5 (4%) would “always” accept a paper based on a Rubriq report. Over half (54%) would consider the report and 42% would always insist on putting the manuscript through their own journal’s peer review process. Based on these general findings, it appears that Rubriq may function to supplement, but not replace, journal-centric editorial and peer review.
- Reviewers have multiple motivations. Of the 172 votes, those who would be willing to review for an external company based their participation on 1) being paid sufficiently for their time and expertise, 2) being guaranteed to receive high-quality, relevant manuscripts, and 3) receiving reputational (non-financial) rewards for their work. Less important was that the company was a non-profit organization.
- Reviewers have two price points. When asked how much money reviewers expected in exchange for providing a review, the results were bimodal. While one-third (33%) of respondents stated that they would not accept financial compensation for reviewing a paper, 73% of those respondents requiring compensation would only review when the rate was at least $100/manuscript, and 41% would only review for at least $200/manuscript.
- Authors want objective, high-quality reviews from experts. Of the 280 votes cast, authors ranked expertise, objectivity, and speed as the most desirable qualities from the review process. Not surprisingly, authors also considered that the service should be affordable (or paid by someone else). While 30 votes were cast for improving the chances of having a manuscript accepted for publication, no votes were cast that Rubriq would guarantee a favorable review. As editors frequently request authors to provide a list of suggested reviewers upon manuscript submission (and often a list of unfavorable ones as well), I find it odd that respondents would not select this choice. Did this question evoke authors’ social-desirability bias?
This straw poll suggests that many editors are hesitant with the notion of outsourcing the peer review process. The results are based on perceptions, however, and perceptions can change when a company can prove itself to deliver results. Remember that editors were also skeptical of outsourcing line editing, layout, and production services when first presented with the option.
Second, the results may suggest two separate review markets — one, a volunteer model based on developing and maintaining trust relationships between editor and reviewer that rewards reviewers with prestige; the other, a commercial model that is based on delivering a valuable service for the right price. Both can coexist in the same market (think the United States Army and private contractors), and each has its benefits and weaknesses. A prestigious society publisher may have no problem soliciting experts in the field for high-quality timely reviews. A commercial archival journal may have a much harder time finding a competent reviewer for its manuscripts. For these publications, a commercial solution may work, especially if the author (or supporting institution) is willing to pay for the services. The journal publishing market is large enough — and diverse enough — to support both models.
If you know of other ways to reward and incentivize the review process, let us know in the comments. And thank you for your participation in the survey.