Science conference season is in full swing, and I can guarantee that an animated discussion of everyone’s most disliked reviewer is going on at every single one. When I was a grad student, everyone lived in fear of Reviewer 3. For some unknown reason (an unhappy childhood, a phobia for science, or perhaps gout), they were often the most negative. Worse still, the editor always agreed with them.
So why would Reviewer 3 get a reputation as the most uncharitable reviewer? There are many possible reasons, but at least one is a legitimate complaint — editors sometimes write an additional anonymous review, typically as the last reviewer on the list, and then use this review as support for their decision. It’s not common, but at least a few editors I’ve encountered (~10%) have thought it was a valid strategy.
It does make sense. The editor has evaluated the other comments and arrived at what they believe is the right decision, but doesn’t feel that the authors will swallow it without additional backing. Since it’s the right decision, why not give your reasoning as an extra ghostwritten review?
The problem here is that the editor is misleading the authors into believing that four independent opinions are behind the decision on their paper, when in fact there are only three — the two real reviewers and the editor twice. Worse still, as the extra review is apparently the basis for the editor’s opinion, it seems to carry more weight in the decision. This is particularly true if the editor thinks the paper should be rejected but the real reviewers were unexpectedly positive, or told the authors they liked the paper but were scathing in their confidential comments (this happens way too much). In either case, the ghostwritten review allows the editor to get to the “right” decision.
This deception matters a lot. If one person doesn’t like your new shoes, you can shrug it off as their bad taste. If two apparently independent people don’t like your new shoes, then maybe purple really isn’t your color. Similarly, the peer review process represents a sampling of the community’s opinion about a paper, and because only two or three opinions are sampled, a single outlier can have a very strong effect. Editors ghostwriting reviews thus generates the impression that a significant proportion of the community (33% or 50%) shares the opinion of the editor, which simply isn’t true.
Rumors of ghostwritten reviews are part of science folklore; as a result, authors are less inclined to believe that a negative decision is a genuine distillation of the reviewers’ and editor’s opinion. It might make for a good conversation at the conference mixer, but shenanigans like this have eroded trust in the peer review system, and have fueled the current crusade for alternative forms of review. Since these alternative review processes are at best untested and at worst just terrible, we have to do all we can to get authors’ trust back.
If we’re going to do away with ghostwritten reviews, we ought to understand when they happen. Typically, the editor was unable or unwilling to return the paper without review but still thinks it must be rejected, so they then need to ensure that the reviews are negative. One route is to send it to reviewers who probably won’t like the paper, and hope that they come through with “reject” recommendations. This approach clearly isn’t a fair or balanced review process, but it still involves gathering independent opinions about the paper (which could even change the editor’s mind). If this strategy fails and the editor only gets positive reviews, they may feel their only way to get the rejection is a ghostwritten review.
The alternative is actually simple — the editor just pastes their detailed comments into the body of their decision, or attaches them as a file, and this eliminates the ambiguity over whose opinion they represent. It’s much more up front, and can generate flak for the editor and the journal, but this approach can be mitigated by a careful complaints procedure and the editor-in-chief being generally supportive of their editors and their decisions. Having decision letters centrally approved by the editor-in-chief also makes ghostwritten reviews easier to catch and provides a chance to discuss the problem with the editor. Lastly, allowing editors to return more papers without review eliminates many of the situations where it becomes tempting.
Who knows, maybe one day we’ll be able to begin the ghostly tale of Reviewer Three with, “Once upon a time, very long ago . . . ”
Discussion
19 Thoughts on "The Referee Who Wasn’t There: The Ghostly Tale of Reviewer 3"
I am trying to get my head around “ghost-written” reviews in a system where reviewers are anonymous. Many editors do not critically read the entire paper. There is no time for that. They have associate editors and reviewers for that. If the editor feels that the reviewers did a crappy job, the editor can assign more reviewers or just review the paper themselves. Depending on their attitude for turn-around time, the editor may choose to read the paper and write the review. Most editor summaries try to referee the reviewer comments for the authors. This is not the same as reading the paper and giving it a critical review.
To me, the bigger problem is when a reviewer has a grad student do the review under the invited reviewers account.
Hi Angela- I mean ‘ghost-written’ in the sense that the authors see the review as written by reviewer X, but the actual author was the editor. Editors certainly can choose to either put a lot of time into evaluating the manuscript themselves or just working from the referee comments, but in the former case they should make it clear that those opinions are their own and not derived from an additional independent person.
But why? Why should the editor dislose that he or she wrote the review. You make it sound like there is some sort of vote tabulation and it’s wrong for the editor to get two votes. I just don’t get the argument here.
You’d agree that it would be wrong for the editor to ghostwrite three different reviews and then use those to support their decision? That’s why it’s also wrong for them to get two regular reviews and then ghostwrite one themselves.
Editors shouldn’t get two votes in the review process because they don’t need them- theirs is always the deciding vote and should be based on the opinions of the external reviewers and their own judgement. Bolstering their decision by also pretending to be a reviewer is unnecessary and dishonest.
Interesting. This is a tactic I employed exactly once during my university press days some years ago. My review wasn’t #3, however. It was review #1. The manuscript in question had already been submitted, reviewed, and rejected previously, having on that occasion been sent to two outside readers and thoroughly reviewed in-house. In truth, it was clear virtually from the outset that the manuscript was hopeless, but the author had a strong local connection, resulting in a decision to give the manuscript the benefit of the doubt first time around. When the author refused to take “no” for an answer and resubmitted the manuscript essentially unchanged, we were in need of a politically acceptable way to turn it down once and for all. My solution, which at the time I thought was original, was to write a review that essentially summarized the previous reviewers’ objections, to which I added my own and was clearly and unambiguously negative, and to forward it anonymously to the author along with a rejection letter. It seems to have worked in that we never saw the manuscript again, nor was our decision challenged by the author’s local supporters. As I say, this was a one-time experience, one I had forgotten about until seeing this piece. And now it turns out not to have been original or unique after all. Great minds thinking alike, or just hard pressed editors needing a practical, politically acceptable, solution to a common publishing problem? If the latter, what are the alternatives?
I can confirm, Tim, that this practice exists and that some editors think it’s acceptable. It isn’t – it’s a deception and unfair – and shouldn’t happen. There should always be transparency, with editors putting their comments and analysis in their decision correspondence under their own name if they do need, for whatever reason – which should be explained – to carry out a ‘review’.
Just like to provide some reassurance about third reviews. Good journals and editors put a lot of thought into sending manuscripts for further review, e.g. when they get differences of opinion in the first set of reviews, one of the first reviews is inadequate, or there are suspicions about fairness. It shouldn’t be an automated process. First thing to do in the case of divided opinions is to check that a reviewer hasn’t reviewed the wrong manuscript or uploaded the wrong review – it happens! Then it’s an intelligent and thoughtful process. Deciding whether the third reviewer should know they’re the third or whether they should assess a manuscript without that knowledge, if they’re to know, should they see the initial reviews or be given any direction or asked to concentrate on specific aspects. It’s also in certain circumstances used as a way to evaluate a reviewer’s report.
All goes to show why good and expert management of the peer-review process is so important.
Thanks Irene- it is worth emphasizing that ghost writing reviews is very rare, and editors almost always give up the practice when they hear that it’s not encouraged at that journal.
To add to your list, one other reason why the last reviewer can be the most negative applies only to resubmissions: the first two reviewers commented on the original version and are now happy, but the third reviewer is new to the paper and has found a whole new set of problems. Authors can get upset when their paper is rejected because a new reviewer didn’t like it, but if reviewer 3 has a valid point and the paper is flawed, it doesn’t make sense to ignore just because the other reviewers didn’t notice it the first time around.
I always wonder what circumstances would require confidential comments to the editor. I’ve reviewed many many papers but I’ve never use that box. In my opinion if it cant be said to the authors then its underhand and runs against ethical principles of peer review. I’m amazed to hear that reviewers can give positive reviews to authors but negative to editors! Where I come from we call that two faced. Dave
We get plenty of confidential comments, ranging from an informal note to the editor because the reviewer knows them personally, to allegations of plagiarism or fraud that they would like us to investigate before making ‘public’ in the decision letter (which gets seen by the authors and all the referees). The case above is often one of degree- the reviewer says what they actually think to the editor, and then soft-peddles it to the authors, perhaps because they don’t want to hurt their feelings.
This is an appalling practice, for the reasons you note, but surely it’s got to be very rare, at least in my field (ecology). I’ve served as a handling editor for years and I’ve never heard of such a thing, or heard anyone complain about it. My own view, and the view of every handling editor I’ve spoken to, is that you only agree to be a handling editor if you’re prepared to put your name on your decisions. I often give authors detailed comments of my own, which amounts to an additional review–but as part of my signed decision letter.
Hi Jeremy- I guess this isn’t something that people ever advertise that they do, and it’s only because I get to see all the decisions that I can catch it. It’s only a small proportion of editors that have asked to do it (under 10%), and then only once (I tell them it’s against journal policy); this is probably an underestimate of its overall frequency because presumably some editors out there are fond of the practice and use it often, and others resort to it when they feel they have to.
Not long ago I had a paper that probably went through such a practice in the journal Molecular Ecology. Two of the three reviewers recommended “Accept with minor reviews” while the third (which was actually Reviewer #1) recommended “Reject”. The Associate Editor followed, not surprisingly, Reviewer #1 and rejected the manuscript. After a few days in dismay, I realized my paper belonged in journal in which the peer-review process would be more objective. Few months later is was published in Plos One. Although I cannot be 100% sure this was a case of ghost-writing, I think that even journals managed by conscientious editors such as Tim, this practice certainly happens.
Thanks for the comment. I’d have to be quite the douchecopter to write a post decrying a practice while still letting it go on under my nose. So, there are two reasons why I know that the above didn’t happen. First, the reviewer order on our decision letters for resubmissions is always fixed so that reviewer #1 (the first from the previous round) is still the first in the resub decision letter, so they would never have been third. This saves endless confusion when discussing the decision letter (e.g. the above).
Second, even if the first reviewer did reject the paper, that was the opinion of a genuine reviewer and not the editor- I don’t let our editors ghostwrite reviews and the decision would never have gone out with a ghost written review in it. I’m glad you managed to get your paper published.
Reblogged this on Virtual & Real World Research – Dr Suzanne Conboy-Hill and commented:
You have to hope transparency would address behaviours of this kind, but it is alarming to think people feel the practice is acceptable. It isn’t. At best, good work doesn’t see the light of day and at worst, life-changing, world-changing, innovation is suppressed. We are better than that, surely?
My role as Editor does not allow me the anonymity of a reviewer, because there is nobody to moderate my comments. Sometimes I or an Associate Editor must give detailed review comments, but I would regard it as fraud to give such comments the status of an anonymous peer reviewer.
The real ghost writers are the postdocs and students who, all too often, are the ones writing the reviews for the Prof. who has more important things to do. In my experience, postdoc written reviews have been sent without even being checked over once. I’d put my money on the third reviewer being an angry postdoc who already has enough to do without taking on yet more work they get no credit for.
Hi Chris- this is an interesting point. We certainly get a few requests from reviewers wanting to share the review with a student (which is fine), but I hadn’t heard that PI’s would farm out the whole review and then claim it as their own. All in all, it would be much easier if they declined the initial invite and suggested that we ask the postdoc instead- the latter would get credit for the review and we’d be able to weight their comments accordingly. More broadly, this is probably a symptom of the uneven distribution of review requests (e.g. here), which we’re making some progress towards solving.
I am a amazed you haven’t heard of this. I don’t think I know a postdoc who hasn’t been asked to review a paper for their PI. If fact in several labs I worked in as a postdoc, it was a running joke that the 3rd review was the ‘angry postdoc’.
I did give the worse case scenario, which I’ve only witnessed on the odd occasion. Mostly a PI will check things over – make sure things have not gone too far. But even so, very rarely will it be a truly combined effort. Mostly, they trust their postdoc’s opinion and don’t actually check that the comments are fair or accurate.
From my interactions with early-career researchers I think this happens more than is generally realised. A few weeks ago I came across a postdoc who had done around 20 reviews for her lab head that had been returned under just his name. Besides being very unfair, it does mean there is a ‘ghost’ reviewer the journal doesn’t know about, and so the usual pre-review checks on potential conflicts, requested exclusions etc, can’t be run, or, as you say Tim, comments weighted appropriately.
It doesn’t allow appropriate credit to be attached to the ECR, and they don’t get to receive any of the acknowledgements, rewards, etc that many journals give their reviewers. Critically, they also don’t get to build up reviewing records with journals. These are important for a number of reasons – being asked to review direct in their own right, provision of letters of confirmation of review when requested, consideration for future editorial board appointments – and may become even more so if moves to have peer-reviewing more formally recognised as a professional activity are successful. I always advise the ECRs to raise the issue with their PIs next time (but not always easy for some), and I tell PIs to make sure they let journals know if they want to involve a junior person and explain why.
