The vast majority of academic authors are supportive of peer review, believe that the process improves their manuscripts and, if done properly to blind identities, helps to minimize discrimination, a recent study reports.
The white paper, “Peer review in 2015: A global view“ was released Friday from Taylor & Francis. In it, readers will find a summary of an online survey and six focus groups. The white paper is accompanied by a supplement that reports individual survey questions and their results.
Comparable to the responses of authors from other peer review surveys (see Sense About Science, 2009, and more recently, NPG’s Author Insights, 2015) authors were largely supportive of the process and conservative in their views. There wasn’t much evidence that the “system is broken,” contrary to the opinions expressed by frustrated academics and tech companies promising to fix the system.
The online survey, sent to Taylor & Francis authors who published in 2013, asked a lengthy set of questions that attempted to discern between experience and expectations. Responses were divided by broad discipline (STM versus HSS), and role (author, reviewer, and editor).
The response rate for these surveys was low–5.5% for STM but 11% for HHS. From the world map on page 4, it is also clear that T&F had a very difficult time getting responses everywhere except North America and Europe. As a result, we should not consider the results to representative of the STM and HSS populations. (Note: The authors of the report confuses statistical precision with sampling representation.)
Respondent fatigue is also apparent as the researchers documented a “sizable, and sudden, drop-off in the response rate” (p.12) midway through the survey. This is not entirely surprising, given the survey length and the cognitive work asked of respondents.
While the questions posed in the survey were clear, I found many of them leading. For example, Q8 asks:
In your opinion, how capable are each of the following types of peer review of preventing discrimination based on aspects of the author’s identity (such as gender, nationality or seniority)?
This question, posed hypothetically, leads respondents to select Double Blind Review–the only form of peer review in the answer list that attempts to conceal author identity. Similarly, the wording of Q9 (delaying publication of a competitor’s work), and Q10 (encouraging favorable reviews) also leads respondents to rating Double Blind Review over all other types of review.
Surveys are extremely difficult to construct and analyze, and I don’t wish my remarks be construed as being entirely dismissive of this report. In spite of its methodological and analytical weaknesses, there is a lot that can be learned from this survey. But just like mom and apple pie, it doesn’t tell us anything that we don’t already know: Most people like their mothers, or at least the idea of motherhood.
The fundamental weakness of the T&F study–and others like it–is that it treats peer review as a concept, rather than a collection of diverse tools used in a varied set of situations for specific purposes. No one would seriously disagree that we need a toolbox, but its the tools we should be evaluating.
Nothing exemplifies this confusion between peer review as a concept versus peer review as a toolbox more than survey question 1, which asks authors the “purpose of peer review” and provides them a list of objectives, from improving the quality of a paper to detecting plagiarism and fraud. This is like asking a carpenter what the purpose of a toolbox should be, by providing him with a list of jobs ranging from hanging a picture on the wall to determining whether the house was wired to meet code.
I was contacted by Taylor & Francis to review this survey for The Scholarly Kitchen, so in a sense, I am participating in post-publication review. I do feel that this report would have benefited greatly had it gone through rounds of pre-publication review — especially by those trained in social science research and statistical analysis — before being disseminated.
If publishers are sincere in their intention to learn about how to improve peer review, the first step is to stop thinking about peer review as a concept and start thinking of it as a toolbox.
While this change in thinking may seem like a simple rhetorical flip, it fundamentally changes the way we pose our investigation, from a marketing approach, “What do authors expect from peer review?” to a scientific approach, “Can we identify specific problems and develop tools to solve them?” Surveys, like T&F’s Peer Review in 2015, looks at peer review from a marketing perspective. Regretably, an approach that allows respondents to rate and rank tools based on their overall popularity is not going to improve our understanding of which tools work best in any particular situation.
Should it matter if screwdrivers get the highest user response, followed by hammers and saws, if your job just requires some spackle, tape, and a putty knife?