External peer review is widely criticized as being too slow, biased, and expensive. For most journals, however, it is a necessary process few editors, authors, or their readers would every consider dispensing with. Despite all the blunt attacks peer review receives in the media, the vast majority of scientists believe that the process improves the quality of scientific papers and should not be replaced. The machine is not broken, but it can be tinkered with.
To date, most attempts at tinkering with peer review have focused on minimizing bias. To this end, several journals have experimented (successfully or unsuccessfully) with implementing some form of open review.
Rubriq, a new venture — covered in a recent interview of co-founder Keith Collier by Michael Clark — attempts to work on the issue of speed. The underlying presupposition of Rubriq is that the process could be sped up by giving reviewers a financial incentive to review quickly. Rubriq also posits that the entire process could be made more efficient if an external company managed the peer review process, rather than being managed on a journal-by-journal basis.
Personally, I am intrigued by the notion of privatizing peer review. Peer review, as it is practiced by academic journals (with few exceptions), relies on a complete voluntary market of experts willing to donate their time because they benefit from the system itself, derive competitive advantage by seeing competitor’s manuscripts, or feel that peer review is a duty and obligation to one’s profession, among other personal and altruistic reasons.
The peer review market doesn’t have to work this way, however. Proposals to pay into the system by reviewing papers and accumulating credits, which can be traded for having one’s own papers reviewed (PubCred), or creating a closed community of peers (Peerage of Science), consider non-financial solutions for speeding up the process and discouraging free riders.
Academic publishing is a very complex market. And like other proposals, the success of Rubriq will depend more on how other parts of the system (authors, reviewers, editors, publishers, funding agencies, academic promotion boards) fit into the solution. Rubriq may succeed as a general solution, or may fill a niche within a neatly defined discipline and addressing its particular needs. Rather than discussing the various benefits and problems of the Rubriq solution (readers can review the excellent dialog with Collier in his interview’s comment section), I thought it might be useful to ask readers of the Scholarly Kitchen to provide some feedback through a few quick poll questions designed to approach Rubriq from the perspective of editors, reviewers, and authors:
Thanks for your feedback. Come back in a day to see how others have voted, and I’ll report the straw-poll results next week.
Discussion
10 Thoughts on "Privatizing Peer Review — A Short Survey"
Interesting post Phil. The survey is a nice touch. I don’t see this as an improvement in the system. It adds a whole new set of costs and associated conflicts of interest. When journals have some real connection to the scholar/researchers they serve through a society or other means, finding reviewers, and reviewers willing to take the time to do a good job is generally not a problem. Unfortunately scholarly publishing seems we are slipping away from this connection.
Dear Phil,
A very interesting survey, and I am looking forward to see the results and commentary.
Thank you for fairly, albeit slightly inaccurately, noticing Peerage of Science too in you blog post (I am one of the founders of the company). It seems your survey questions are exclusively about Rubriq though, rather than about companies offering solutions to peer review in general. I am not certain what privatizing exactly means in this context (we do not charge authors or pay the reviewers), but we do offer a paid service: journals can purchase the right to use Peerage of Science and make direct publishing offers through it.
Janne, thanks for the response. Perhaps a better term would be “third-party” or “extra-publisher service”. I noted Peerage of Science in the piece because there are alternatives to creating a private company to provide these services. However, including two financial transactions in the equation (author => company => reviewer) does make Rubriq very different from Peerage and changes the costs, motivations and rewards dramatically. To me, it made sense to include Rubriq in the survey questions so respondents have a clear idea of what I’m asking. Best, Phil
Interesting investigation: I wonder if there is scope also to extend the questionnaire to publishers (as well as authors, reviewers and editors)? Strikes me that some publishers, big or small, might be interested in partnering with these services.
Peer review is used well beyond reviewing papers for publication, from grant proposals to post-publication letters and more. There has been considerable study of it in information science, sociology, and medicine.
In my review a few years ago, I was surprised to learn (“Undermining peer review,” Society 38, 2 (J/F 2001):47-54; “Peer review,” in Higher Education in the United States. An Encyclopedia, 2002.) to what extent it has been under attack. Quality is shortchanged while some projects are moved along or tanked. As an editor I have also witnessed referees attempting to torpedo competitors or alternate theories, the sort of thing that can give the process a bad smell.
Only a few days ago I wrote about “Peer review and the world banking crisis” see http://journalofpathology.wordpress.com/category/general/
We may be approaching an Academic Liquidity Crisis and I fear that Rubriq as suggested will not help!
🙁
I think these questions are really insightful and hit on many of the key strategic assumptions that require market validation for Rubriq. I’m sure we can learn from the responses as we continue to think about and evolve our business model and strategy.
I’m assuming people will take the survey and then read comments; so don’t expect (or want) to skew the results of the survey. But I do want to use the forum to clarify our current position related to these questions.
Q1 For Editors: Would you accept a paper based on a report from Rubriq?
– We do not ever expect Rubriq to fully replace the peer review that happens in traditional journals. Perhaps a better question, at least for traditional journals would be “Would you expedite a paper that was submitted with a Rubriq report?” Our value for traditional journals is to expedite the process by suggesting the appropriate journals based on the Rubriq ratings, subject area, and author preference. We believe the Rubriq report will be a tool to supplement the peer review that happens within the journal, not a replacement.
– For Mega-OA (Gold) or Green OA, we believe we can provide a valid science stamp. Other benefits for mega-OA include the initial stratification of papers in the “big heap” based on the Rubriq article level metrics. It allows for a more efficient and organized approach to “publish then filter” for the literature and provides a starting point for post-publication peer review.
Q3 For Reviewers: l would review a manuscript if I were paid at least…
– Reviewers who do not want to (or can’t for various reasons) receive payments can waive their fee. Rubriq isn’t a non-profit, but we have applied to be a B-Corp and are looking for ways to put these funds back into science.
Q4: For Authors: I would consider submitting my manuscript to Rubriq if… (check all that apply)
– I truly hope the option of “I am guaranteed a favorable review” receives zero responses. If someone uses Rubriq hoping to be guaranteed a glowing review, they are going to be disappointed. Our credibility as an independent organization will depend on our accurate and unbiased assessment of the research.
One overlooked reward for reviewers is a reference letter. Reviewers often request one for immigration or tenure. Our ScholarOne system lets me look up a reviewer’s reviews and give a detailed evaluation of his or her contribution, all based upon solid evidence. As I advise, “If you did good work for us, you will like this reference!”
I suppose a system like Rubriq, which would incorporate a reviewer’s work for multiple journals, would allow an even better evaluation of their qualifications.
I think this approach might work better for some mass-driven journals like PLOS One than for journals where reviews need to be more extensive and numbers of submissions are lower. I wonder how journal editors would view farming out this crucial part of their role to another party. As an acquiring editor for scholarly books, I would be very loath to consider any such arrangement as I feel that the choice of readers has a lot to do with the overall integrity of a publishing program that results in helping create an identity and reputation for a publisher’s list. Perhaps attitudes differ between journal and book editors on this score.
