butterfly collectionPeer review is not a single organism, but a diverse ecosystem, in which many species can compete, coexist, or mutually support each other.

I want to revisit the genus of portable peer review, a model in which peer review is conducted outside the auspices of the journal. We’ve covered portable peer review over the past several years, by interviewing its founders, exploring incentives and credit for peer review, and covering failed attempts.

This post will briefly focus on three companies that currently provide portable peer review: Rubriq, Axios Review, and Peerage of Science. I am conspicuously avoiding covering companies that manage the peer review process for publishers, consortia of journals that agree to transfer manuscripts and reviews, as well as cascading peer review within a single publisher. This is a scholarly publishing blog, so you are likely to find good coverage of these topics in other posts. And last, my purpose is not to provide an exhaustive review — you’ll find extensive details on each company’s website — but to provide you with the salient differences in how these three services conduct portable peer review.

Rubriq is built on a service model that matches manuscripts with registered reviewers. Since 2012, Rubriq has completed 1,688 article reviews. The company has little information on the fate of the papers that were reviewed, however. According to Damian Pattinson, VP of Publishing Innovation at Research Square, about half of the submissions were from authors themselves, the other half were from publishers. Pattinson does not know how many of the papers were re-reviewed by the publisher — either as a standard procedure, or because the publisher was conducting an evaluation of Rubriq quality — nor does he know the number of papers that have been published. Pattinson explained that Rubriq has been expanding its services for publishers, providing certification that manuscripts have been checked for plagiarism, figure manipulation, and quality of English, among other factors. Prices for reviews start at $600 USD, with reviewers receiving a flat fee of $100. Rubriq is a division of Research Square, located in Durham, North Carolina.

Axios Review is built on an agent model that reviews manuscripts, matches them with appropriate publication outlets, and contacts journal editors on the authors’ behalf. Because of their active role in the review and referral process, Axios has collected extensive data on their performance. According to Tim Vines, Managing Editor and Founder of Axios Review, Axios has reviewed 420 papers to date, 130 (31%) of which are known to have been published (see list). The remainder of the papers are in process of being revised by the authors or submitted to a journal. A small fraction of manuscripts have likely been abandoned, meaning the authors have stopped trying for publication. Of the 130 published papers, 65 (50%) were re-reviewed by the journal; the other 50% were accepted without further review. 85% of papers reviewed by Axios are accepted for publication by the first journal. Axios charges authors $250 USD which is due upon receipt of reviewer comments. Some publishers (chiefly BMC) deduct this fee from their article processing charge. Editors and reviewers are not paid for their work. Tim Vines reports that submission numbers have been growing at 30% per year. Axios Review is a registered non-profit company operating in British Columbia, Canada.

Peerage of Science is built on a community model, where authors are in charge of the peer review process. Unlike Rubriq and Axios, Peerage of Science does not solicit reviews on the authors’ behalf, but leaves the process up to their community of registered peers. As the author specifies the time period under which specific tasks must be accomplished, the number of manuscripts initially submitted to the system gets whittled down at each stage. According to Janne Seppänen, Co-Founder, and Managing Director of Peerage of Science, 364 manuscripts have been submitted to the system since 2012, 21% of which have failed to receive a single review. Authors can also terminate the review process at any stage, so the number of manuscripts that get reviewed, revised, received a final evaluation, and exported to a participating journal is about one-third (35%) of initially submitted manuscripts. The success rate drops to 28% of papers that receive a publishing offer or an invitation to submit directly to the journal. Peerage of Science posts a list of known published papers, although Janne does not know whether the journal put these papers through additional rounds of peer review. Their business model is supported by publishers, who pay €220 Euros for each peer review process leading to publication. Submissions since 2012 have been relatively constant, with an uptick in 2016. Peerage of Science is a for-profit company registered in Finland.

Portable peer review has had working models since 2012. Taken together, the number of manuscripts flowing through these services is tiny compared to the traditional model of managing peer review in-house. As detailed above, giving authors the freedom of managing or porting their reviews comes at the expense of gathering good data. Axios takes an active role in the review and referral process, while Peerage leaves most of the management details up to the author and reviewer community. Rubriq falls somewhere in the middle. Axios is supported by charging authors, while Peerage is supported by charging publishers. Rubriq appears to be shifting from author to publisher-side charges.

Among the three companies, only Rubriq reviews can be considered completely portable. Working on behalf of the author, Axios’ agent model limits author choices, although it appears to be highly effective in matching manuscripts with prospective journals. Similarly, Peerage of Science’s business model requires publishers to pay for review transfers, so portability is limited to a group of journals who have bought into the Peerage system.

Small markets coupled with the diversification of the leading company (Rubriq) into services aimed at supporting publishers may indicate that portable peer review has limited overall reach. Nevertheless, like book publishing agents, Axios’ agent model may have tapped into a real need to provide author services that do not stop with peer review. Peerage of Science, in contrast, gives authors the tools they need to be agents of their own manuscripts.

Journals have a history of outsourcing parts of its publication process to external service operations who can do it faster, cheaper, or better than keeping the process in-house. So, why hasn’t peer review been outsourced? There is far more diversity in how journals conduct peer review than, say, page composition or XML conversion. Perhaps it is this ecosystem diversity that has prevented portable peer review from taking off?

Phil Davis

Phil Davis

Phil Davis is a publishing consultant specializing in the statistical analysis of citation, readership, publication and survey data. He has a Ph.D. in science communication from Cornell University (2010), extensive experience as a science librarian (1995-2006) and was trained as a life scientist. https://phil-davis.com/

Discussion

8 Thoughts on "Whither Portable Peer Review?"

Phil, in Peerage of Science 70% of manuscripts get through to the final stage:
– for 16% the authors don’t take action after it,
– 30% are exported by authors,
– plus 24% got direct publishing offers or submission invitations from participating journals.

Authors have full “portability” freedom in Peerage of Science, they are not limited to participating journals. Only participating journals can make offers during the process, but authors can choose to reject the offers and create an export link and give it to any outside journal they choose (free of charge for both author and for receiving journal).

How does one calculate “success rate” for peer review?

Above, and often in questions I encounter, for peer review services it is weirdly and frustratingly equated with number of accepted articles divided by submission volume. But people don’t say Nature or Science have a “success rate” under 10% for their peer review process, do they?

When a careful peer review blows a bad paper to smithereens, so that editors who see those reports never touch it, that event is a success for all journals who thereby avoid Retraction Watch, and for science and society at large, right? And when a hastily scribbled peer review by an authority leads a journal to publish a paper that should not have been published, that is a failure, right?

Perhaps difficult to estimate, but I might advocate a “success ratio” instead: the ratio of true positives (manuscripts that should be accepted and are accepted) + true negatives (manuscripts that should be rejected and are rejected), to the false positives (manuscripts that should have been rejected but got accepted) + false negatives (manuscripts that should have been accepted but got rejected).

I think it’s clear that these services are addressing a real problem, as most journals reject well over half their submissions. Given the time and effort involved in getting a paper ready and then submitted to a journal, and then the time lost while waiting for a decision, submitting to the wrong journal should be a real pain point for authors.

The community certainly rushed to adopt PLOS ONE as an outlet when it became clear that they offered both high acceptance rates and a decent Impact Factor, and researchers also seems willing to pay 5-10 times more than e.g. Axios costs in APC’s.

Whether or not authors become increasingly willing to pay for independent peer review and connection to a journal should become clear over the next 1-2 years: either one of us will take off, or we won’t.

Until journals accept these portable (independent) peer review reports on par with their own peer review process, it is going to be a hard sell, whatever the price point. And current prices are set so low that they need a lot of volume for the company to profit, improve, and diversify. So, it seems like the first step is to convince publishers that companies like your can do it as well, or better, and/or cheaper then them. Nature Communication tried an experiment with Rubriq, but the scientific community pushed back and independent peer review was scrapped. The experiment may have been more readily accepted if it weren’t pitched as a fast-track service for those willing to pay, however. Taylor and Francis has been offering expedited review and publication services for over a year now and I haven’t heard any push-back against it.

The rates T&F are charging for expedited review and publication are pretty hefty – eg $8500/€6250/£5500 for a 10-page FastTrack published article. I can’t find any details on how these fees are apportioned. Does anyone know if the reviewers – who are a critical element in the promised timeline being achieved – are paid? Or if they receive any other sort of remuneration to incentivize them to take part? Any details about author take up?

In Peerage of Science, the price for authors is zero, and there is decent level of acceptance of the concept among journals (now 67 with direct access + all of Springer in export destinations), and most direct offers to authors in Peerage of Science state there is no need for additional round of peer review. So it is largely “on par”. Yet, we are not drowning in submissions after five years, though recent growth is encouraging.

The thing is, for journals and publishers independent peer review is only a pragmatic decision, but for authors, a submission decision is a matter of habit and tradition and emotion and willingness to try new things with precious fruits of their work. For a typical manuscript, four or five authors need to get convinced to try something new, and some of them are understandably thinking nothing but how that submission affects their chances to survive the tenure track race, while others are complacent to continue doing everything the way that led them to their tenure. Not the easiest crowd to encourage to revolution, even where all other players welcome them to.

It is a hard sell, even at price point zero, not because journal acceptance would be missing, but because academia is conservative by nature.

Comments are closed.