Without a doubt, the number one complaint at every editorial board meeting I attend is how to find good, qualified reviewers. As more and more manuscripts come in, the pool of tried and true reviewers is being bombarded with requests from multiple journals. Journals and books aren’t the only culprits. Many of the top experts being called on to review manuscripts are also being asked to review grant proposals. As has been discussed, a lot, this is an unpaid, extracurricular activity for which no real credit is given to the reviewer. Anecdotally, we hear that reviewer fatigue is the main issue with reviews not being completed or invitations being ignored or declined. A recent study took a look at what happens at one journal when a reviewer declines.

An editor at the American Political Science Review, Marijke Breuning, and the journal staff compiled reviewer data to determine why people decline to review and also if there are any differences in reviewer behavior by gender. In 2013, the journal sent out 4,563 requests to review either new submissions (96%) or revisions (4%). The good news was that 82.8% of reviewers responded one way or another to the invitation. In a close community, as you would find with a society journal, reviewers will have the courtesy of responding to their colleagues.

Almost 60% of the requests with responses were positive with a decline rate of 23%. As most journal editors can tell you, getting someone to accept the invitation to review is not the same as actually getting the review. Some of the reviewers in the study were dismissed because the number of required reviews was satisfied or they just never completed the review.

The authors used their manuscript submission system to analyze the reasons given for not accepting the invitation to review. They suspected that reviewer fatigue was to blame. Here is what they found:

• Almost 29% of the reviewers gave no reason for declining the invitation to review
• 25% said they were too busy but did not indicate that having lots of papers to review was the culprit
• 14% did decline with an excuse that they had too many other review invitations
• 33% of the reviewers declined for reasons such as “not an expert” (8%), “on leave” (3.3%), “already reviewed this paper” (3.2%), “university admin duties” (3%), etc.

Interestingly, 28 people declined and said they did so because they are editors at other journals. Mostly these editors referred to their workload with their own journal essentially taking themselves out of the “pool of experts.”

The authors saw no statistical difference in the behavior of male vs. female reviewers; however, the reasons for declining certainly gives us a glimpse that is familiar. More men declined for purposes of having administrative duties (such as department chair or dean) at a higher rate than women did. Likewise the bulk of the journal editors declining were men. Women were more likely to decline due to being on “personal leave or sabbatical” and “personal issues” such as family member’s illness. Also not surprising is that there were more women declining due to “maternity or paternity leave.”

There has been much debate about wasted reviewer time. A single paper could be reviewed by 6-10 (could be more) people at multiple journals before ever getting accepted. Without a doubt, we have more papers being submitted today than we did 10, even 5, years ago.

Every board meeting I attend has a discussion about reviewers—not fast enough, not responsive to the invitations, not writing quality reviews, refusing to re-review revised papers, etc. There is definitely a feeling that there aren’t enough “good” reviewers to go around. These anecdotes may not be telling the whole story.

Using data from our tracking systems is one way to determine whether reviewer fatigue is real and whether editorial boards are exacerbating the problem. Most, if not all, tracking systems have reports showing the number of reviewers used in a given time period compared to the number of reviewers in the database. Likewise, reports are available to see how many papers per reviewer are being assigned and a “top reviewer” report that shows how frequently the same people are used over and over again.

On occasion, an editor asks us if they can create a custom list of “preferred reviewers.” They understandably don’t want to sift through 10,000 names and they don’t want to take the time to look at the review history of each person before choosing them. But a preferred list is really only adding to the fatigue problem. Selecting 50 “go to” reviewers out of 10,000 means that you are burning out those 50 awesome reviewers.

Breuning et al. include some tips for avoiding reviewer fatigue:

1. Search beyond the usual suspects. Many reviewers are invited based on their reputation or who the editor knows personally.
2. First time reviewers are more likely to accept the invitation, even if they need a little coaching to get it done. Expanding your search for reviewers to databases of dissertations and conference programs may help.
3. Personalize your invitation letters where possible.

In the end, only 14% of the reviewers who declined said that the reason was due to having too many review requests. Certainly there are other time-consuming pressures on the people journal editors count on as reviewers.

I would be very interested in seeing stats from other journals who may be collecting this kind of information. Assuming there is a problem is one thing. Identifying the problem with data provides a much better story.