Finding willing academics to review a manuscript is getting more difficult for some journals. Unfortunately, this difficulty may also be negatively affecting the judgement of journal editors, a new study reports.
The paper, “Difficulty of recruiting reviewers predicts review scores and editorial decisions at six journals of ecology and evolution”, was published in the October issue of Scientometrics. Its author, Charles Fox is a professor of evolutionary and behavioral ecology at the University of Kentucky and the current editor of Functional Ecology.
For his study, Fox analyzed peer review data on nearly 52 thousand reviews for 24 thousand research papers that were sent out for review in six separate journals.
He reports that the more difficulty editors had in obtaining reviews on a manuscript (measured by the proportion of review invitations that were accepted), the less likely the manuscript was invited for revision. Put another way, papers that were ultimately rejected experienced more difficulty finding willing reviewers.
Moreover, the difficulty editors experienced finding reviewers predicted manuscript scores. Less trouble: higher scores; more trouble: lower scores.
These results are not altogether surprising. Editors routinely send title, author, and abstract information to invited reviewers, who use that information before committing to review. As a reviewer, it’s not hard to glean from the abstract whether the paper is relevant, important, and ultimately worth your time.
Now, here is where the study gets interesting. The ultimate decision to accept or reject a paper was not fully explained by what reviewers thought of the manuscript. Even after controlling for reviewer assessment, recruitment difficulty still predicted editors’ decision. Fox postulates that declining to review a manuscript sends a subtle message to the editor that the paper is not worth publishing. The more reviewers who decline, the stronger the negative signal. He writes:
[E]ditors may be biased, although subtly, against papers for which they have difficulty recruiting reviewers, either because they believe that difficulty recruiting reviewers is more informative about manuscript quality than it actually is or because they become annoyed at or frustrated by such papers.
Fox has no data to test his annoyance/frustration hypothesis, although anecdotally, it seems to ring true. Contacted for comment on his paper, Mark Johnston, Editor-in-Chief of the journal GENETICS, offered this response:
Editors (myself included) do get frustrated when they have to send out more and more invitations to review, but my impression is that they put that frustration aside once the reviews are in hand. […] Of course editors are human, so it would not be surprising if they had some residual unconscious bias induced by the frustration they experienced, but if that’s the case it seems they do a pretty good job of squelching that because its effect is small.
While Fox studied 6 ecology and evolution journals, the results are likely generalizable across other disciplines. They may not be applicable to journals that reject a sizable proportion of papers without undertaking external peer review, however. For high-profile journals that only send out papers that stand a good chance of being published, recruiting suitable referees is usually not a problem.
Tim Vines, a former managing editor of ecology journals and someone who has done research on the peer review process, agrees. He writes:
[E]ditorial rejection is a key journal tool: papers that are unlikely to be accepted take up disproportionate amounts of editorial effort, so sending them back to the authors straight away saves everyone a lot of hassle.
For journals that do not require manuscripts to report novel or significant findings, locating a willing reviewer may be much more difficult. Commercial options, where authors pay to have their paper independently reviewed have failed to gain market acceptance, however. For the foreseeable future, frustration may continue serve as a signal for many journal editors.