Peerage of Science (www.peerageofscience.org) was founded in 2011 by Janne-Tuomas Seppänen, Mikko Mönkkönen, and Janne Kotiaho. Based in Finland, the company seeks to improve peer review by establishing a more transparent and accountable process that also provides more freedom to both authors and reviewers.
Peerage of Science is among several new ventures working in this area (readers may also be interested in this recent interview with Keith Collier of Rubriq).
The following interview with co-founder and managing director, Janne-Tuomas Seppänen, was conducted via email.
Q: What is Peerage of Science?
A: It is a new kind of peer review system, and a radically new kind of submission system.
Q: Who is Peerage of Science?
A: Over 1500 scientists around the world, currently mostly from biological sciences, but the community is open to any qualified scientist. The founders include myself, Mikko Mönkkönen, and Janne Kotiaho. I am an Academy of Finland-funded post-doc and Mikko and Janne are professors at the University of Jyväskylä.
Q: What problem did you set out to solve?
A: Several interconnected problems, but most importantly these two:
First, the quality of peer review in the traditional system is, well, variable. Even with the most prestigious journals, the author is too often left wondering whether the reviewer spent any time thinking about the work. There is an element of lottery in the process of getting published and hence in the visibility of results and resulting career and research opportunities. We (scientists and publishers, as well as taxpayers funding all this) really should strive to reduce the lottery element as much as possible.
Second, the time from first submission to eventual acceptance somewhere was actually shorter in Darwin’s time than it is now. Many unnecessary delays and inefficiencies in the traditional system impose costs on everyone. Scientists lose time, and sometimes funding, as a crucial paper lingers in purgatory. Publishers lose money. Society at large loses the valuable effort of some of its most expensive minds, and knowledge of new science is delayed.
Q: What led you to start the company?
A: I was a young postdoc agonizing about my contract ending soon and approaching a funding application deadline. I had an article ready with exciting results, and knew it would change the fate of my applications. But the paper was sitting in peer review with and had been for over a year. The paper eventually did get published in Proceedings B [which was not the journal that had been sitting on it] and my research did get funding on the next round, so things worked out for me personally.
But after reading studies about publishing delays and the poor predictive success of peer review, I realized what I experienced was not just bad luck; everyone (including publishers) faces all sorts of costs from the quality issues and inefficiencies of the legacy system. And really the key thing missing was a method to provide academic recognition for peer review work well done (and punishment for sloppy, unqualified or biased work), plus a method for journals to efficiently consider manuscripts concurrently and make publishing offers (tailored where necessary) to any manuscript they wish to publish.
“Someone should start a service doing that” was a frequent comment I heard when discussing these ideas. Instead of waiting for someone else, we decided to just do it.
Q: How does it work?
A: From publisher’s point-of-view:
The journal editor has access to the peer review system and can follow anything potentially interesting. If the editor finds the manuscript is not suitable for the journal, they simply stop following it. If at any point of the peer review process the manuscript proves to be something the journal wants to publish, the editor can send a private publishing offer to authors. Journals are free to define the terms of their offers (e.g. requiring revisions, or even additional review). Authors are free to accept or decline.
From scientist’s point-of-view, the peer review process goes through four stages:
1) Upload and Essays. The author uploads the manuscript and defines a deadline for each stage. Peers find the manuscript via customizable alerts or tweets, and any non-affiliated, qualified Peer (we check identity and qualifications of everyone) is free to engage to review. Reviews are structured “Essays,” plus categorical revision recommendations.
2) Peer review of peer review. The peer reviewers read each other’s essays and evaluate each other.
3) Revision. Authors revise the manuscript, and write a reply to reviewers.
4) Final evaluation. Peer reviewers evaluate the revised manuscript both by scoring standard aspects such as methods, inference, writing etc. and by writing a final statement directed more towards editors who may be tracking the process.
Q: What happens if a paper does not get any reviews in the specified time period?
A: The author gets a polite email saying the process was terminated because the work did not attract any reviews. Typically this happens when authors uploaded the manuscript just before going to field-season or holidays. It is not a good idea to submit in mid-December. Failing to get peer reviews is quite rare though: so far 91% of submitted manuscripts have received at least one peer review.
Authors are free to start again (one example got four reviews on second try, and a direct publishing offer), though of course it is a good idea to consider first whether there was something wrong with the manuscript, or if the short blurb-message going out with alerts should be modified. With freedom in reviewer engagement, the number of reviewers engaging is in itself one measure of the quality of the manuscript.
Q: How does the “peer review of peer review” work?
A: Each reviewer is required to give three scores to each other reviewer: 1) how accurate and justified are arguments about the merits of the manuscript, 2) how accurate and justified are arguments criticizing the manuscript, 3) how well does the reviewer discuss the implications of, or improvements for, the research.
Q: What is the PEQ?
A: PEQ stands for Peerage Essay Quality. It is the sample size weighted (using the number of reviewers in a given process) average over all the peer review quality scores a Peer has accumulated. PEQ shows up in Peer’s profile and we also provide a widget Peers can embed on their own website or online CV to display current peer-review performance. In addition to the grand average, the Peer’s profile and the widget also show PEQ calculated for particular fields.
Editors can see anonymized profiles of reviewers in processes they follow, even when the Peer chooses to not disclose identity, so editors can inspect past performance of reviewers.
Q: Are reviewers anonymous to the author? To other reviewers? To editors?
A: Reviewers are free to choose if and to whom their names are displayed. Authors and editors have the same freedom. The default setting is anonymity. We think triple-blind – where everyone’s judgment of scientific arguments is free from perceptions of institutional or personal prestige – would be great, but complete transparency has its merits too. So it is up for everyone to decide themselves, and let community standards evolve.
Q: Why would a reviewer want to participate, what is the incentive?
A: A big motivation is reciprocal service to one’s field of science. In Peerage of Science, reviewers can choose manuscripts where their expertise matters most, at a time that suits them, and their reviews are used by several journals, instead of being discarded if single journal rejects a submission. Reciprocity is enforced too: reviewing increases your Reviewing Balance while your submissions lower it, and if your Reviewing Balance is negative you cannot submit your own manuscripts to Peerage of Science.
An important new motivation is participating in measuring and improving the quality of peer review via peer review of peer review. Scientists are evidently eager to do that – other tasks are done just before deadline, but the peer review of peer review task gets completed, on average, in less than two days.
Peerage of Science also wants reviewers to have an entirely selfish motive to do their very best in peer review. The PEQ scores accumulate in each Peer’s profile, and can be displayed in his or her own website or online CV. The best reviewer is recognized annually with a Reviewer Prize. Also, a service called Peerage of Science Commissions relays paid peer review assignments (e.g. in grant proposal or conference abstract evaluation), and we naturally recommend reviewers having high PEQ scores and essay counts to customer organisations. Finally, we hope someday soon publishers interviewing prospective editors will ask “What peer reviewed evidence do you have about your ability to judge other people’s research? What is your PEQ?”
Q: How is Peerage of Science different from other peer review services like Rubriq and Axios Review?
A: There are three key differences:
1) Peer review of peer review. This is a central feature of Peerage of Science process, designed for measuring and improving the quality of peer review. I am not aware of other services employing this process.
2) Freedom and control. Authors are free to define deadlines and to decide whether and what revisions are warranted based on reviews; to choose from sometimes multiple publishing offers; or decline all offers and submit anywhere they want with a link to Peerage of Science reviews included. Reviewers are free to engage work they are interested in. Editors are free to consider any manuscript, not just those authors choose to submit to them, and are free to define the terms of each offer as they wish. Everyone is free to choose the level of anonymity desired. Other services are more traditional, in being more editor-controlled and having more limited freedoms.
3) Fees. Peerage of Science is “free” for scientists (though of course their work is the contribution that makes the service possible, so scientists do pay in-kind for the services they receive). Many other services charge a fee from authors, so authors purchase peer review and associated services, and publishers get their service for free.
Q: What is the business model? Who pays?
A: Journals purchase a service from Peerage of Science. There are two options to choose from:
Full Access –Provides the journal with the ability to participate in peer review processes, and the ability to make offers at any time. Publisher pays, but only in instances where authors accept journal’s offer.
Connect –Places clickable journal logos inside the author’s export-menu in Peerage of Science (shown after peer review), allowing direct one-click submission from Peerage of Science to the journal. Publisher pays a flat monthly fee regardless of the number of submissions thus received.
A publisher can choose to have Full Access for some of their journals and Connect for other journals.
Q: Why would a publisher want to participate? What is the benefit?
A: A participating journal gets an opportunity to make an offer before outsiders even see the manuscript. Imagine an author of a ground-breaking paper, faced with a choice between one click to acceptance in your journal versus uncertain fate after lengthy additional wait at a more prestigious journal outside (and your journal can spice its offer with, say, “will be featured on the cover”). Chances are good your journal snatches a high-impact article otherwise destined for elsewhere.
Participating in a system that promotes freedom for everyone and seeks to measure and improve quality of peer review, sends a strong positive signal about values to the scientific community. Participating says that the publisher cares about, respects and promotes the integrity and transparent evaluation of the quality of peer review, and the freedom of authors and reviewers – so much that it is willing to actually pay for a dedicated service, making all this possible not only for itself but for all others too.
Q: Can publishers publish the reviews from Peerage of Science along with the papers, much as eLife is now publishing comments from their review process?
A: If they get permission from the reviewer, then yes.
Q: How is Peerage of Science an improvement over traditional, journal-based peer review?
A: Peerage of Science does not see itself doing “decoupled” or “portable” peer review. Peerage of Science is a platform for community-based peer review and that community includes participating journals.
The only “decoupling” occurring is that a journal participating in Peerage of Science claims no exclusive right to dictate who gets to peer review what, no exclusive right to utilize the peer reviews provided by scientists, and no exclusive right to accept or decline a submission.
A journal not participating in Peerage of Science usually does claim all those exclusive rights over scientist’s work – if you are a scientist, you perhaps should pause to think about that before submitting your next paper to a traditional process, and before accepting the next traditional reviewing request. Do you think those claims are justified? Do you want to support them?
Q: Can publishers use Peerage of Science in place of their own peer review processes?
A: Yes, and we wish they do so. Several papers have been published already via direct offers in Peerage of Science where journals did not need any additional in-house peer reviews.
Due to the freedom in reviewer engagement and transparent peer-review-of-peer-review, we trust Peerage of Science will establish a widely recognised standing of highly trustworthy peer review. Journals are free to make offers that are conditional on additional in-house peer review, but that will hopefully be rarely necessary.
Furthermore, journals can solicit reviewers to a particular manuscript under peer review in Peerage of Science just as they would in their own process, and editors can freely participate in the process discussing matters with authors and reviewers if they choose to. Journals can also upload manuscripts to Peerage of Science themselves (with author’s consent, of course). The journal can get an exclusive offering right for such manuscripts, for a fee. A portion of the exclusive rights fee is returned if the journal later releases the manuscript to be free to receive offers from other participating journals.
Q: What publishers are you working with?
A: BioMed Central, Pensoft Publishers, PeerJ and PLOS. In addition, a number of journals from Wiley, Springer, and some societies are using Peerage of Science on a trial basis. And a few journals are listed as “welcoming links”, meaning they encourage authors to include a link to Peerage of Science peer review process when submitting to them, which is a free service for both the authors and the journal. Journals are listed at www.peerageofscience.org/journals/
For more on Peerage of Science, please see the FAQ page on the company’s website.
Discussion
4 Thoughts on "An Interview With Janne-Tuomas Seppänen, Co-founder of Peerage of Science"
I think that Peerage of Science and PLoS ONE are the most innovative ideas in academic publishing since we abandoned the hard copy submissions. Other advantages of Peerage of Science include solving the “Tragedy of the reviewer commons” and giving the opportunity to students engage in the peer review process (which is often confined to a small group of colleagues of the editorial board of a journal). These early career students/scientists can learn how to write good peer reviews by doing it.
To fully realize the potential of Peerage of Science we have to imagine a world where no other peer review system exists, only PoS. Thousands of journals would be part of it and pay a small fee every month. Millions of scientists would be part of the platform and each manuscript would be reviewed by several reviewers, but only once (not multiple rounds of review/rebuttal/decision in different journals). If an editor wanted to be sure a manuscript be reviewed by a particular expert in the field, he could just invite the expert to be part of the process.
I just hope the service gains more traction in the next years than it has so far. Just over 1500 people engaged in an universe of 6-7 million scientists is far less support than it deserves.
Thanks Paulo. A world with a single system of anything is probably a dangerous thing to even imagine. Even if the system was founded and led by such utterly flawless people as in this case…
Seriously though, I could see a future where
– peer review cooked by journal staff alone is increasingly seen as an inferior way to run a scholarly journal, used only by a few small publishers who can’t or do not want to invest money in “proper” peer review.
– almost all serious peer review gets done via a moderate number of peer review systems similar to Peerage of Science (with variation and competition in scope, focus, pricing, etc like any healthy industry) and journals then make choices in which systems they want to (and can afford to) maintain presence in. Small specialist journals choose one small specialist community, large general journals purchase rights to several large general systems to cast a wide net for best work. And it is still probably cheaper for each than the legacy system was.
It’s not clear to me how the reviews become useful to publishers unless they know the reviewers’ identities, and disclosure of that information appears to be purely voluntary on the part of reviewers. But I do, in general, like the concept of PoS and hope that it gets extended to the social sciences and humanities.
Reviewer’s reviewing history (“reputation”) is displayed in the profile even when reviewer chooses to remain anonymous. And so far reviewers have been willing to disclose name upon request in almost all cases.
While I do understand the rationale in what you say, isn’t there a risk of (unconscious) bias if reviewer’s identity influences how editor perceives his or her arguments? Peer reviews that in effect say: “This should not be published, and I do not need to justify that statement further because I am the premier authority on this subject” are, sadly, not unheard of.