Science conference season is in full swing, and I can guarantee that an animated discussion of everyone’s most disliked reviewer is going on at every single one. When I was a grad student, everyone lived in fear of Reviewer 3. For some unknown reason (an unhappy childhood, a phobia for science, or perhaps gout), they were often the most negative. Worse still, the editor always agreed with them.
So why would Reviewer 3 get a reputation as the most uncharitable reviewer? There are many possible reasons, but at least one is a legitimate complaint — editors sometimes write an additional anonymous review, typically as the last reviewer on the list, and then use this review as support for their decision. It’s not common, but at least a few editors I’ve encountered (~10%) have thought it was a valid strategy.
It does make sense. The editor has evaluated the other comments and arrived at what they believe is the right decision, but doesn’t feel that the authors will swallow it without additional backing. Since it’s the right decision, why not give your reasoning as an extra ghostwritten review?
The problem here is that the editor is misleading the authors into believing that four independent opinions are behind the decision on their paper, when in fact there are only three — the two real reviewers and the editor twice. Worse still, as the extra review is apparently the basis for the editor’s opinion, it seems to carry more weight in the decision. This is particularly true if the editor thinks the paper should be rejected but the real reviewers were unexpectedly positive, or told the authors they liked the paper but were scathing in their confidential comments (this happens way too much). In either case, the ghostwritten review allows the editor to get to the “right” decision.
This deception matters a lot. If one person doesn’t like your new shoes, you can shrug it off as their bad taste. If two apparently independent people don’t like your new shoes, then maybe purple really isn’t your color. Similarly, the peer review process represents a sampling of the community’s opinion about a paper, and because only two or three opinions are sampled, a single outlier can have a very strong effect. Editors ghostwriting reviews thus generates the impression that a significant proportion of the community (33% or 50%) shares the opinion of the editor, which simply isn’t true.
Rumors of ghostwritten reviews are part of science folklore; as a result, authors are less inclined to believe that a negative decision is a genuine distillation of the reviewers’ and editor’s opinion. It might make for a good conversation at the conference mixer, but shenanigans like this have eroded trust in the peer review system, and have fueled the current crusade for alternative forms of review. Since these alternative review processes are at best untested and at worst just terrible, we have to do all we can to get authors’ trust back.
If we’re going to do away with ghostwritten reviews, we ought to understand when they happen. Typically, the editor was unable or unwilling to return the paper without review but still thinks it must be rejected, so they then need to ensure that the reviews are negative. One route is to send it to reviewers who probably won’t like the paper, and hope that they come through with “reject” recommendations. This approach clearly isn’t a fair or balanced review process, but it still involves gathering independent opinions about the paper (which could even change the editor’s mind). If this strategy fails and the editor only gets positive reviews, they may feel their only way to get the rejection is a ghostwritten review.
The alternative is actually simple — the editor just pastes their detailed comments into the body of their decision, or attaches them as a file, and this eliminates the ambiguity over whose opinion they represent. It’s much more up front, and can generate flak for the editor and the journal, but this approach can be mitigated by a careful complaints procedure and the editor-in-chief being generally supportive of their editors and their decisions. Having decision letters centrally approved by the editor-in-chief also makes ghostwritten reviews easier to catch and provides a chance to discuss the problem with the editor. Lastly, allowing editors to return more papers without review eliminates many of the situations where it becomes tempting.
Who knows, maybe one day we’ll be able to begin the ghostly tale of Reviewer Three with, “Once upon a time, very long ago . . . “
I am trying to get my head around “ghost-written” reviews in a system where reviewers are anonymous. Many editors do not critically read the entire paper. There is no time for that. They have associate editors and reviewers for that. If the editor feels that the reviewers did a crappy job, the editor can assign more reviewers or just review the paper themselves. Depending on their attitude for turn-around time, the editor may choose to read the paper and write the review. Most editor summaries try to referee the reviewer comments for the authors. This is not the same as reading the paper and giving it a critical review.
To me, the bigger problem is when a reviewer has a grad student do the review under the invited reviewers account.
Posted by Angela | Jul 31, 2012, 9:21 amInteresting. This is a tactic I employed exactly once during my university press days some years ago. My review wasn’t #3, however. It was review #1. The manuscript in question had already been submitted, reviewed, and rejected previously, having on that occasion been sent to two outside readers and thoroughly reviewed in-house. In truth, it was clear virtually from the outset that the manuscript was hopeless, but the author had a strong local connection, resulting in a decision to give the manuscript the benefit of the doubt first time around. When the author refused to take “no” for an answer and resubmitted the manuscript essentially unchanged, we were in need of a politically acceptable way to turn it down once and for all. My solution, which at the time I thought was original, was to write a review that essentially summarized the previous reviewers’ objections, to which I added my own and was clearly and unambiguously negative, and to forward it anonymously to the author along with a rejection letter. It seems to have worked in that we never saw the manuscript again, nor was our decision challenged by the author’s local supporters. As I say, this was a one-time experience, one I had forgotten about until seeing this piece. And now it turns out not to have been original or unique after all. Great minds thinking alike, or just hard pressed editors needing a practical, politically acceptable, solution to a common publishing problem? If the latter, what are the alternatives?
Posted by Irving Rockwood | Jul 31, 2012, 9:36 amI can confirm, Tim, that this practice exists and that some editors think it’s acceptable. It isn’t – it’s a deception and unfair – and shouldn’t happen. There should always be transparency, with editors putting their comments and analysis in their decision correspondence under their own name if they do need, for whatever reason – which should be explained – to carry out a ‘review’.
Just like to provide some reassurance about third reviews. Good journals and editors put a lot of thought into sending manuscripts for further review, e.g. when they get differences of opinion in the first set of reviews, one of the first reviews is inadequate, or there are suspicions about fairness. It shouldn’t be an automated process. First thing to do in the case of divided opinions is to check that a reviewer hasn’t reviewed the wrong manuscript or uploaded the wrong review – it happens! Then it’s an intelligent and thoughtful process. Deciding whether the third reviewer should know they’re the third or whether they should assess a manuscript without that knowledge, if they’re to know, should they see the initial reviews or be given any direction or asked to concentrate on specific aspects. It’s also in certain circumstances used as a way to evaluate a reviewer’s report.
All goes to show why good and expert management of the peer-review process is so important.
Posted by Irene Hames (@irenehames) | Jul 31, 2012, 11:15 amI always wonder what circumstances would require confidential comments to the editor. I’ve reviewed many many papers but I’ve never use that box. In my opinion if it cant be said to the authors then its underhand and runs against ethical principles of peer review. I’m amazed to hear that reviewers can give positive reviews to authors but negative to editors! Where I come from we call that two faced. Dave
Posted by Dave Allsop | Jul 31, 2012, 7:02 pmThis is an appalling practice, for the reasons you note, but surely it’s got to be very rare, at least in my field (ecology). I’ve served as a handling editor for years and I’ve never heard of such a thing, or heard anyone complain about it. My own view, and the view of every handling editor I’ve spoken to, is that you only agree to be a handling editor if you’re prepared to put your name on your decisions. I often give authors detailed comments of my own, which amounts to an additional review–but as part of my signed decision letter.
Posted by Jeremy Fox | Jul 31, 2012, 9:42 pmNot long ago I had a paper that probably went through such a practice in the journal Molecular Ecology. Two of the three reviewers recommended “Accept with minor reviews” while the third (which was actually Reviewer #1) recommended “Reject”. The Associate Editor followed, not surprisingly, Reviewer #1 and rejected the manuscript. After a few days in dismay, I realized my paper belonged in journal in which the peer-review process would be more objective. Few months later is was published in Plos One. Although I cannot be 100% sure this was a case of ghost-writing, I think that even journals managed by conscientious editors such as Tim, this practice certainly happens.
Posted by Reply to Jeremy Fox | Aug 6, 2012, 12:56 pmThanks for the comment. I’d have to be quite the douchecopter to write a post decrying a practice while still letting it go on under my nose. So, there are two reasons why I know that the above didn’t happen. First, the reviewer order on our decision letters for resubmissions is always fixed so that reviewer #1 (the first from the previous round) is still the first in the resub decision letter, so they would never have been third. This saves endless confusion when discussing the decision letter (e.g. the above).
Second, even if the first reviewer did reject the paper, that was the opinion of a genuine reviewer and not the editor- I don’t let our editors ghostwrite reviews and the decision would never have gone out with a ghost written review in it. I’m glad you managed to get your paper published.
Posted by Tim Vines | Aug 6, 2012, 1:10 pmReblogged this on Virtual & Real World Research – Dr Suzanne Conboy-Hill and commented:
You have to hope transparency would address behaviours of this kind, but it is alarming to think people feel the practice is acceptable. It isn’t. At best, good work doesn’t see the light of day and at worst, life-changing, world-changing, innovation is suppressed. We are better than that, surely?
Posted by Suzanne Conboy-Hill | Aug 1, 2012, 11:45 amMy role as Editor does not allow me the anonymity of a reviewer, because there is nobody to moderate my comments. Sometimes I or an Associate Editor must give detailed review comments, but I would regard it as fraud to give such comments the status of an anonymous peer reviewer.
Posted by Ken Lanfear | Aug 1, 2012, 7:39 pmThe real ghost writers are the postdocs and students who, all too often, are the ones writing the reviews for the Prof. who has more important things to do. In my experience, postdoc written reviews have been sent without even being checked over once. I’d put my money on the third reviewer being an angry postdoc who already has enough to do without taking on yet more work they get no credit for.
Posted by Chris | Aug 2, 2012, 11:07 am