Believe it or not, the current structure of journal peer review represents an optimal solution to multiple conflicting demands: reviewers should be anonymous to encourage objectivity, but someone needs to vouch for their expertise to the authors (one of the reasons we have Editors). Busy researchers rarely volunteer to do full length reviews, so someone (the Editorial Office) needs to directly invite them and keep them on track to deliver their comments in reasonable time. Having 4 or more reviews generates a lot of revision work for the authors and takes up too many reviewers’ time for just one article, so journals aim for 2-3 reviews, and so forth.
However, the current structure may only be a ‘local optimum’ – there may be a radically different but superior system out there, but we can’t reach it by tinkering with the current system. Bold experiments with peer review are therefore very welcome, as one of these might just hit upon a new and better approach to research evaluation.
One such bold experiment is Select Crowd Review, which is implemented by chemistry publisher Thieme and the commenting platform Filestage.io. Their project recently made it to the finalist stage of the ALPSP Award for Innovation in Publishing, so I got in touch with Niklas Dorn (the CEO and Co-Founder of Filestage) to find out more.
What drove Thieme and Filestage to experiment with crowd review?
The idea for Expert Crowd Review came from Prof. Benjamin List (Max Planck Institute for Coal Research). He was not satisfied with the speed of the peer review process and was looking for ways to significantly reduce review time. At the same time, he was trying to streamline the review process and simplify reviewing for reviewers.
Ben List’s idea basically was to leverage the intelligence of the crowd to review scientific papers faster. So instead of sending out a manuscript to just 2 reviewers in peer review, it would be sent to a crowd of selected reviewers that would then review the document all together at the same time. This would speed up the review process significantly because in a larger crowd of 20 reviewers there would always be enough reviewers available to review within a short timeframe.
After learning about this idea the Thieme Group started a collaboration with the online proofing specialist Filestage. Filestage then built a secure and encrypted crowd review solution and ran multiple proofs of concept with Thieme.
Could you tell us briefly how it works?
Sure, the expert crowd review solution is a secure software that runs in the browser. There is no download nor installation required.
- Editors will share manuscripts via a secure and encrypted review link with a selected group of reviewers in the browser.
- Reviewers review, comment, and approve a manuscript anonymously in the browser. Reviewers collaborate seamlessly and discuss different view points together.
- Editors have a clear overview of the review and approval status of their manuscripts at every time. The editors will see if a manuscript has been approved by the crowd or rejected with change requests.
- All comments from the crowd can be exported as one PDF and sent back to the authors for further improvements.
How are reviewers selected and invited?
The editors select the reviewers within their own database. Once the reviewers have been selected they can be invited in two ways for crowd review.
The first option is to invite reviewers via a protected review link that allows them to directly access a file, review, and approve it. The editor can share this link via email or via their own reviewer management systems.
The second option is to add reviewers with their email address to your specific expert crowds in Filestage. Once a reviewer is part of an expert crowd he/she will get an automatic invite if new manuscripts within their domain are waiting for review. Editors can nurture their crowds and add or remove reviewers anytime.
What’s a typical timeline for expert crowd review?
Reviews are completed within 72 hours instead of months before in the peer-review process.
How do the reviewer comments differ from those obtained through standard peer review?
Control experiments showed that the crowd collectively provided more-comprehensive feedback than in peer review. There are real discussions between the reviewers for the first time since they review all together in the same document. So reviewers actually reported that reviewing feels less like work because there is a real scientific discussion happening. I think herein lies a big chance for the scientific community: to have more interaction in the review process.
Are there any empirical comparisons of the two approaches?
Synlett and Prof Benjamin List have A/B tested the two approaches simultaneously. So, the same paper was reviewed in expert crowd review but also in a traditional peer-review process. Crowd review was completed a lot faster. Taken together, responses from the crowd showed at least as much and often more, attention to fine details, including supporting information outside the main article, as did those from conventional reviewers.
You can find the detailed article on the Nature blog.
Does the short timeline exclude reviewers who think more deeply about the article?
No, so far we have not seen that. If a reviewer decides to participate in crowd review then she/he has typically 72 hours time to review. This is more than enough for most of the reviewers since they rarely spend more than a few hours to review one single manuscript.
What other effects do you see?
Aside from much, much faster, better and more pleasant for all involved? Lightened workload for the editorial office, which does need to send endless streams of invitations to review, followed by ever more strident reminders to deliver.
Why do you think that other areas haven’t adopted expert crowd review?
We decided to keep our crowd review solution exclusive to hand-picked journals and publishers for the start. For us and Thieme it was important to learn from the editors and the crowd reviewers first what works well and what could be improved. With that knowledge, we have further developed the functionality of the crowd review solution.
How can a journal participate in crowd review?
We are now slowly opening the crowd review platform for more journals. Interesting journals and publishers can apply for expert crowd reviews via our website to run their own proof of concept.
Discussion
26 Thoughts on "The Curse of Reviewer 89: An Interview with Filestage’s Niklas Dorn"
So if I follow, for the trial from 100 reviewers participating, dozens participated in each review. Seems like that would work out to a lot more reviews per reviewer in the long run. The novelty would likely wear off. But what if after the initial crowds thin, maybe only 3-5 reviewers out of the many invited would tackle papers of particular interest? Even then, having concurrent, shared reviews where reviewers can see each other’s comments and can communicate among themselves about the paper would lead to better reviews. And quick. This is the best idea for improving the process I’ve heard. Good luck with it!
My understanding is that the time investment per reviewer is much lower, in that they’re able to agree with previous comments (rather than write out the same thing from scratch) or just focus on the part of the manuscript where they feel best able to comment.
The “wisdom of the crowd” theory is based on the independent assessment of knowledgeable individuals. I would be concerned that the first reviewer would set the tone and agenda for evaluation of the peer group. For example, what if the first reviewer, a senior researcher in the field, writes a glowing review, says the work is novel and important, and wants a quick acceptance with few changes. Would that affect the views of everyone who comes in afterword? Similarly, say that senior researcher has a long-lasting grudge with the senior author of the manuscript, completely pans the paper, calls it unconventional, critiques the methodology and statistical analysis, and argues for a rejection. I can’t imagine it would be easy for a postdoc to argue that the paper should be accepted.
And lastly, isn’t this model similar to the group evaluation model used by eLife? It seems like we can learn a lot from them, even if they don’t use the same software.
This was exactly my thought on reading this. I really like the collaborative nature of the idea, but how does one prevent “groupthink” and encourage independent/original critiques in a system like this?
Hi Phil, the behaviour of groups in a crowd review process would be certainly a great topic to study.
One thing we have done to prevent the crowd from being intimidated of titles and status of other individual reviewers was to make reviews anonymous. That was quite an effective way. Furthermore there are always editors that keep an eye on the reviews and can moderate, if the feedback is very one-sided.
True, but simply reading a manuscript closely and critically takes time, even if a full written report isn’t needed.
That was my thought as well — the timesink here (at least in my experience) is in carefully reading the article and then thinking about it, not writing up the peer review comments/report. Asking 15-100 people to put in that same amount of careful reading time seems less efficient than having 2 or 3 people do it.
It is definitely not disallowed for our referees to read manuscripts closely and take their time. In fact, they often do that. However, nobody has to spend weeks on a manuscript if he or she doesn’t want to or doesn’t have the time. In fact, only a fraction of the crowd needs to comment.
Thanks a lot for your comment Chris Mebane! Yes, the great thing is that this works also with smaller crowds. So you don’t need 50 reviewers to review every time. A pool of 15+ reviewers per crowd is ideal, and it is often enough to have 5+ reviewers reviewing, discussing, and commenting on a manuscript. The interaction between them makes the big difference.
Has anyone experimented with an idea like this for book peer review, especially in the humanities? Our system is broken too, or at the very least in dire straits.
Another question: in book peer review it’s far easier to get prompt reviews if an academic ‘star” is the author and almost impossible for an author with a revised dissertation. I wouldn’t be surprised to learn that this reality crosses all publishing categories and would be interested to learn whether this differential showed up in crowd review.
Hi Wendy, thanks a lot for raising this question. That’s a very interesting thought. We have not started any projects for book reviews, yet, but I would be happy to give it a shot. If you’re interested in a chat to figure out options let me know.
I don’t have data about the “star” factor, but I can imagine that it plays a role. Maybe the guys from Thieme or Emerald can help with that.
Thanks for the great interview and the very thoughtful comments. We have just published an (expert crowd reviewed) account on this story that can be found here: https://www.thieme-connect.com/products/ejournals/abstract/10.1055/s-0040-1705955. All questioned raised here are answered there. One comment on the concern by Phil Davies: You raise an important and valid point that I would like to address. First of all, we do see intense discussions among crowd members and typically biased comments are immediately corrected. Clearly, the crowd does not just follow opinion leaders but can be rather critical. Secondly, we actually do like the consensual way the crowd operates – makes our life as editors much easier. Finally, if you really insist on an independent assessment by the referees, there are technical solutions. Consider for example a delay for the crowd to see each others comments. This could be either a certain time that has to pass, or a certain number of comments, before everybody can see everything.
Question — are the identities of your reviewers made clear to the other reviewers? I’ve heard anecdotal evidence that the community peer review process at the other journal mentioned in this thread can, at times, devolve into a big shot in the field bullying less prestigious reviewers into following their opinion. So it does happen, though I can’t speak to the frequency.
Our referees don’t know each other’s identity. To be honest, I think for quality peer review the authors need to be anonymous. Exactly because we want to avoid situations like you describe.
This should read: “To be honest, I think for quality peer review the referees need to be anonymous.”
1) I see a great potential in this system for training younger reviewers. 2) I was wondering: have you thought of including authors in the discussion at some point? It makes sense sometimes for the authors to clarify their position or to start working on point they see emerging from the discussion. 3) Finally: isn’t this the same as what happens on pre-print servers?
It’s different from preprint servers in that they’re inviting qualified reviewers with a formalized process for reviewing, rather than just leaving things to chance to see who happens to come along any leave a comment or tweet about the preprint. We know that the vast majority of papers on preprint servers never receive any comments, so as we’ve seen over and over again, without editorial oversight driving the process, peer review doesn’t generally seem to happen.
Including the authors in the entire process is an awesome idea! In fact, we are already thinking about this format, which would make our life as editors even easier: just let the crowd and authors sort things out 🙂
In my view, this will definitely come. However, we currently don’t want to overwhelm everybody involved in scientific publishing with too many innovations at the same time…
Your other question concerning the difference to the preprint system has already been well explained below by David Crotty. Thanks for this. I would like to add that pre-print publishing and crowd reviewing are complementary. Because of its incredible speed, crowd reviewing, in some cases, may actually make pre-print publishing obsolete. Why publish something that has not been reviewed if you can get a real and fully peer reviewed paper pretty much in the same time. Of course this will depend on the amount of revisions that are required, which can slow things down significantly.
Slow speed of reviews CAN be a problem but it is not a universal problem across all journals. Under the current system, a few reviewers produce reviews in 30 days or so. The crowd review concept produces reviews in 3 days. How is the fate of science publishing substantially improved by a difference of approximately 27 days? Is there an imperative for science to move at that speed?
Hopefully, the 30-day reviews are not what folks are concerned about. I think they are concerned about poorly managed journals that drag out reviews much longer. With those journals, isn’t the main problem editors who are poor managers? Sending invitations to reviewers and being organized is about good management skills in editorial offices. So editors who can’t manage the current system well are going to be in charge of managing crowd reviews with many more reviewers in much more disorganized chat rooms. Seems designed to create more complaints about the peer review system.
Good point, if it is not broken don’t fix it! Crowd reviewing may not be something for every Journal in every scientific field, especially if you don’t mind the tardiness of the process and there may be good reasons to not minding it. In our area of chemical synthesis and catalysis time is often of the essence, and my impression is that this is also true in other scientific fields.
However, crowd reviewing is not just about speeding up the refereeing process; I would argue that crowd reviewing can also significantly improve it. At least for us at Synlett it works great and more than half of our manuscripts are currently handled like this. But hey, it’s a free country and nobody is forced to using this format, neither our authors nor anybody else in scientific publishing (even if I think that those badly managed journals with lazy editors you mention would actually benefit…).
I’m a psychatrist. Don’t know much about your field. Why is time more of the essence in chemical synthesis and catalysis time more than in other fields?
Of course, because we are in a hurry to change the world as quickly as possible. Just kidding. In reality, in chemistry and other natural sciences there’s always an intense competition of the top groups to discover things first. I know this is a bit of a silly game. To a large degree excellency in science is ego driven. In my field, patenting also can play a role.
Very nicely done experiment, thank you for sharing.
This reminds me of what we are doing at PREreview, except that we do it on preprints. As part of our effort to increase participation in peer review, we organize live-streamed preprint journal clubs, topic-centered interactive preprint journal clubs that are live-streamed via Zoom. These calls are facilitated by us and promote active participation by researchers who join from all over the world to collaborate on improving a preprint, build their network, and meet globally-renowned experts. The output of these virtual journal clubs is a full review generated by the discussion with input from every participant. In some cases, we had the authors participating, and that was a big success!
I wonder if you would be interested in trying that format as another form of open, collaborative peer review. I truly believe that with the right facilitation, there is no need to be anonymous and having a live discussion actually foster constructive criticism and opens up opportunities for collaborations.
This is a beautiful idea! I imagine this could be a great teaching tool for advanced students and postdocs. I feel that it is time to open up peer review to the younger generation. I have often asked my senior postdocs to write refereeing reports for me (when it was allowed by the journal) and then gave them the credit. Often they know more about one area than I do. Expanding our referee pool would also definitely be required to scale-up your reviewing model.
Thank you, Ben. I’d love to discuss this more with you. We are all about empowering the next generation of peer reviewers. Check out more info about our peer review mentoring program and live-stream preprint journal clubs.
Feel free to email us at contact@prereview.org. I’d love to set up a chat.
Thieme & Filestage will host a free webinar about “Select Crowd Review” December 7th, 2020.
Feel free to participate and ask all your questions there.
https://www.thieme.de/de/thieme-chemistry/webseminar-select-crowd-review-161372.htm
