Editor’s Note: Joe Esposito pointed out earlier this week that information is not a zero sum game, “Different metrics measure different things for different audiences.” That said, there is a point where too many metrics can muddle the picture we’re trying to see. “Because it can be measured,” is not a good enough reason for a metric to draw one’s limited attention. Five years ago I suggested using a panel of metrics to compensate for the flaws in each individual metric, a suggestion that has recently been revived by a SPARC white paper. At the time I urged caution: “As we assemble this panel though, it’s important that we don’t muddy the waters further with the impact equivalent of chart junk. Just because you can measure something, it doesn’t automatically result in meaningful data.”
Do we need a metric for everything? Many activities of a researcher are done as a service to one’s community, not out of an expectation that they will lead to financial reward or career advancement. Do we really need to turn philanthropic volunteerism into a carefully tracked and rated competitive exercise? In light of that, and the recent new round of funding announced for Publons, I wanted to revisit the question of how carefully we need to track and rate peer review activities. Is this added complexity necessary when a simple yes/no checkbox would suffice?
Offering career credit to researchers for performing peer review seems like a no-brainer, right? Peer review is essential for our system of research, and study after study confirms that researchers consider it tremendously important. Funding agencies and journal publishers alike rely on researchers to provide rigorous review to aid in making decisions about who to fund and which papers to publish. On the surface it would seem to make sense to formalize this activity as a part of the career responsibilities of an academic researcher. But as one delves into the specifics of creating such a system, some major roadblocks arise.
One such problem falls into the realm of volunteerism and motivation. Right now, most academics see performing peer review as a service to the community. It’s important to the advancement of the field and so they volunteer their time. If instead we turn peer review into a mandatory, career requirement that is rewarded with credit, it changes the nature of the behavior. If we set standards (you must do X peer reviews per year) people will then work to those standards rather than the more generous acts we see today, where good samaritans (and good reviewers) take on much larger workloads.
Economists suggest that incentives (a form of reward) changes motivation, some of which will be actualized by real behavioral change. Educator Alfie Kohn talks about how behaviors change in light of offering rewards in one of his books on parenting:
…there are actually different kinds of motivation. Most psychologists distinguish between the intrinsic kind and the extrinsic kind. Intrinsic motivation basically means you like what you’re doing for its own sake, whereas extrinsic motivation means you do something as a means to an end — in order to get a reward or avoid a punishment…extrinsic motivation is likely to erode intrinsic motivation…The more that people are rewarded for doing something, the more likely they are to lose interest in whatever they had to do to get the reward.
Plugging peer review into a rote system of requirements may threaten both the participation levels and the rigor and enthusiasm with which many approach the task.
A second problem with peer review credit schemes (and perhaps I’m arguing against my own interests here) is the increased power it places in the hands of publishers and editors. Researchers have little control over whether they are asked to do peer reviews. If your career is dependent upon getting a journal to ask you to review a paper, what happens if you’re not on their list? An early career researcher who is not well-known in their field is at an automatic career disadvantage under such a scheme. Many are already resentful of the “kingmaking” ability of editors of journals like Science, Nature and Cell. Turning over even more power over academic career success to publishers might not be so well-received.
Perhaps the biggest problem of all comes when we ask the simple question that must always be asked when new changes to the academic career structure or the scholarly publishing ecosystem are proposed: Who cares? Not “who cares” as in “peer review is unimportant and no one should care about it”, but “who cares” as in “who exactly are we asking to grant credit here?”
As we are constantly reminded, the two things that matter most to academic researchers are career advancement and funding (and the more cynical among us suspect that the former is primarily dependent on one’s ability to secure the latter).
If I was an administrator at a research institution, I’m not sure I’d want my researchers spending an enormous amount of their time helping to improve the papers of researchers at other institutions. If I was a particlarly wise administrator able to see the big picture, I would understand the value of peer review and how it is necessary for the advancement of knowledge. So I’d know some amount of credit is due. But it’s not the primary reason I hired those researchers, nor is it something I want them spending a lot of their time doing. Their job is to do original research.
Similarly, a funding agency gives a researcher a grant to do research. A diabetes foundation is looking to fund research to cure the disease and likely wants fundees spending their time doing original research, not reviewing papers from other researchers. How much should they reward fundees for doing something other than what they’ve been funded to do? And back at that research institution, if much of the tenure decision (at least in the sciences) is based on how much funding one can bring in, then if peer review doesn’t bring in funding, it won’t matter all that much.
How much career credit should a researcher really expect to get for performing peer review? I suspect that at best it will be a few small percent of the overall picture, more likely a box on a checklist – did you do any peer review? If yes, then you get a small bonus amount of credit. No one is going to get hired, tenure or funding based on a stellar record of peer reviewing lots and lots of papers. There’s a different job where you get rewarded for that — it’s called “editor”.
The proposed peer review credit systems currently under examination, both commercial and community-based seem like overkill. Many systems offer extensive tracking, point systems and review of reviewers which may be unnecessarily complex for a yes/no question. As Joe Esposito has trained us to ask, this is at best a feature, certainly not a product nor a business.
Perhaps something along the lines of the work ORCID and CASRAI are doing will suffice in the end. Tag the activity to the researcher’s identifier and offer a simple yes/no or a tally of peer review events for the year. Do we really need anything more than that?