Science conference season is in full swing, and I can guarantee that an animated discussion of everyone’s most disliked reviewer is going on at every single one. When I was a grad student, everyone lived in fear of Reviewer 3. For some unknown reason (an unhappy childhood, a phobia for science, or perhaps gout), they were often the most negative. Worse still, the editor always agreed with them.
So why would Reviewer 3 get a reputation as the most uncharitable reviewer? There are many possible reasons, but at least one is a legitimate complaint — editors sometimes write an additional anonymous review, typically as the last reviewer on the list, and then use this review as support for their decision. It’s not common, but at least a few editors I’ve encountered (~10%) have thought it was a valid strategy.
It does make sense. The editor has evaluated the other comments and arrived at what they believe is the right decision, but doesn’t feel that the authors will swallow it without additional backing. Since it’s the right decision, why not give your reasoning as an extra ghostwritten review?
The problem here is that the editor is misleading the authors into believing that four independent opinions are behind the decision on their paper, when in fact there are only three — the two real reviewers and the editor twice. Worse still, as the extra review is apparently the basis for the editor’s opinion, it seems to carry more weight in the decision. This is particularly true if the editor thinks the paper should be rejected but the real reviewers were unexpectedly positive, or told the authors they liked the paper but were scathing in their confidential comments (this happens way too much). In either case, the ghostwritten review allows the editor to get to the “right” decision.
This deception matters a lot. If one person doesn’t like your new shoes, you can shrug it off as their bad taste. If two apparently independent people don’t like your new shoes, then maybe purple really isn’t your color. Similarly, the peer review process represents a sampling of the community’s opinion about a paper, and because only two or three opinions are sampled, a single outlier can have a very strong effect. Editors ghostwriting reviews thus generates the impression that a significant proportion of the community (33% or 50%) shares the opinion of the editor, which simply isn’t true.
Rumors of ghostwritten reviews are part of science folklore; as a result, authors are less inclined to believe that a negative decision is a genuine distillation of the reviewers’ and editor’s opinion. It might make for a good conversation at the conference mixer, but shenanigans like this have eroded trust in the peer review system, and have fueled the current crusade for alternative forms of review. Since these alternative review processes are at best untested and at worst just terrible, we have to do all we can to get authors’ trust back.
If we’re going to do away with ghostwritten reviews, we ought to understand when they happen. Typically, the editor was unable or unwilling to return the paper without review but still thinks it must be rejected, so they then need to ensure that the reviews are negative. One route is to send it to reviewers who probably won’t like the paper, and hope that they come through with “reject” recommendations. This approach clearly isn’t a fair or balanced review process, but it still involves gathering independent opinions about the paper (which could even change the editor’s mind). If this strategy fails and the editor only gets positive reviews, they may feel their only way to get the rejection is a ghostwritten review.
The alternative is actually simple — the editor just pastes their detailed comments into the body of their decision, or attaches them as a file, and this eliminates the ambiguity over whose opinion they represent. It’s much more up front, and can generate flak for the editor and the journal, but this approach can be mitigated by a careful complaints procedure and the editor-in-chief being generally supportive of their editors and their decisions. Having decision letters centrally approved by the editor-in-chief also makes ghostwritten reviews easier to catch and provides a chance to discuss the problem with the editor. Lastly, allowing editors to return more papers without review eliminates many of the situations where it becomes tempting.
Who knows, maybe one day we’ll be able to begin the ghostly tale of Reviewer Three with, “Once upon a time, very long ago . . . “