While much of the contemporary debate around peer review focuses on both journal articles and STEM fields, here we are going to focus on humanities and social science (HSS) fields where both longer articles and book-length projects are more common. Does HSS peer review have the same functions, goals, and challenges as STEM peer review? My conversation below with fellow Scholarly Kitchen Chef Karin Wulf and Mary Francis, Editorial Director at the University of Michigan Press, addresses these questions and looks forward to the ways in which HSS peer review might evolve.
Alison Mudditt: Karin, as both a working scholar and someone who oversees a substantial publication program, what are the key ways in which HSS peer review differs from STEM journals models?
Karin Wulf: A key point to make right off the bat is that many more, if not most, HSS fields are “book fields”, disciplines in which scholars typically publish their most significant work in book form rather than in journal articles. Journals still play an important role for HSS; HSS articles tend to be longer, and HSS journals tend to publish a small percentage of submissions. Peer reviewers for HSS journals are thus doing pretty extensive work on lengthy submissions, and mostly for authors whose work won’t be published (at least not in the first journal to which it’s submitted). Generally, though, peer review is valued on both sides of the experience. It’s simply part of the culture for authors and reviewers to participate in the intellectual exchange that takes place in the context of peer reviewing. It is also the case that a lot of articles are developed in the context of a book in progress and that peer reviewers for journals thus have always played an important role in moving a book to eventual publication.
Alison: Beyond the common gatekeeping role, does peer review play a different role in these fields?
Karin: Another important point is one we make in Scholarly Kitchen posts a lot — there is tremendous diversity in the culture and practice among and within disciplines, not only between HSS and STEM. It would be hard to generalize about the differential role of peer review within the cultures of each. In many fields different versions or pieces of a project are often shared to solicit feedback from experts. The difference with formal peer reviewing is that the expertise is solicited (by the journal editor) and usually that it is anonymous to each party (double-blind). The value of double-blind review has been debated quite a bit, but I still come down on the side of anonymity as the best hedge against bias.
Alison: Are there distinctions between HSS book and journal peer review?
Karin: Again, there is significant diversity among and within HSS fields that’s important. But in the aggregate, peer review in journals versus books can be quite distinct. Journal editors tend to commission more reviews for each article than do book editors; reader reports for journal articles also tend to be proportionally longer. Journal editors mostly deal with submissions that come in over the transom and are fully responsible for the substantive responses to reader reports, often writing lengthy letters to authors to interpret the readers’ reports and deliver their decision.
But acquisition editors for university presses are both doing more active recruitment and generally less substantive editorial work with their authors. The role of the review itself is therefore different. Books tend to get fewer reader reports (usually two), though often at two different stages, the original and final manuscript submissions.
What peer reviews of journal article and book manuscripts share is basic assessment: how fresh is the research? How persuasive is the argument? What does this work contribute to the field and/or fields? Peer reviewers are asking these same questions of the two different genres.
Alison: Mary, as a seasoned university press editor, what do you think is distinctive about university press peer review?
Mary Francis: As Karin notes, different peer review practices tend to attach to disciplinary norms and product type, more than type of publisher. University presses, even relatively small ones, produce a wide range of publications: scholarly monographs, journals, textbooks, critical editions of primary sources, guidebooks, regional titles, poetry, and more. While all of these are peer reviewed, one of the distinctive features of university press peer review is how it accommodates such a range of publications. When I was working with the Association of American University Presses (AAUP) acquisitions committee to create their new guidelines, there were many discussions of how to zero in on the most rigorous practices that all book peer review ought to share, and the most salient variations had to do with disciplines being considered.
Alison: Peer review has always been at the heart of university press publishing: why did AAUP decide to create a “best practices” document at this point in time?
Mary: The biggest reason is to call attention to the value that peer review provides. The bulk of the document is a detailed ‘how-to,’ which should make clear how many carefully considered steps and decisions are involved as well as how much expertise is required. Editors play an essential role in framing peer review for their constituents within their overarching project of cultivating intellectual endeavors in a given field, matching the needs of a project to the expertise of peer reviewers, and making sure the final product has the greatest possible reach and influence. At a time when higher education and the core values of scholarly research are under intense public scrutiny, the document was designed to advocate for the value of peer review to those whose support is essential: deans and provosts, foundations and funders, regents, elected representatives.
Alison: What processes shaped its development?
Mary: The creation of the document was an immensely rewarding two-year odyssey of discussion and debate, informed, naturally, by rigorous peer review. The AAUP Acquisitions Committee wrote – and debated — the document jointly. The first draft was reviewed by the AAUP board, then revised and submitted to twenty-five peer reviewers, including fellow university press editors, colleagues from museum publishing, commercial scholarly publishing, and university press publishers in the UK. The revised version was the focus of a very lively AAUP Collaboration Lab at the 2015 AAUP annual conference, and generated more refinements leading up to the release of the complete document this past spring.
Alison: The voices demanding alternatives to blind and double-blind review have risen alongside greater experimentation with new models of long-form content, especially those in digital form. I’m interested as to why AAUP chose not to discuss those new models here – was there a reason for this?
Mary: Alternative forms of peer review, and peer review of multi-modal scholarly outputs were very much part of the committee’s discussions: we talked about how multi-modal scholarly outputs challenge publishers to think carefully about parameters that are easier to take for granted in a world based on print books: version control, preservation, citability, discoverability, etc. In the end, the committee felt that there isn’t enough consensus around peer review of alternative forms (or alternative forms of peer review) for there to be a clear set of best practices that AAUP could endorse at this point in time.
Alison: What about reviewer selection? The guidelines highlight the reviewer’s potential to judge the scholarship and argument presented, but I wonder how both of you think about diversity – race, gender, geographies, and other identities – as you select reviewers? Given the challenges the academy faces in increasing diversity, how do we ensure that peer review doesn’t contribute to privileging certain established voices over others?
Karin: A lot comes down to the press director and their expectations of editors. We don’t want conferences to have “Hasslehoff panels”; similarly, we want review panels to reflect diversity of perspectives. Diversity includes race and gender as well as junior and senior scholars, and scholars from different types of institutions; reviewers shouldn’t all come from R1 universities, for example. But here’s another thing — at least in HSS, editors should want as reviewers not only the scholars whose work is exactly spot on the topic, but what our journal editor might describe as a 360 degree view from different field specialties that have a bearing on the work in question.
Mary: We did talk about how “established voices” manifest themselves in different disciplines. Editors’ desire to diversify beyond “established voices” is about exciting, innovative work that goes beyond familiar methodologies and paradigms, and for this editors are often seeking younger scholars who are thought leaders in their field — a pool more likely to manifest a greater degree of diversity. But as more academics around the world hold contingent positions that make it difficult for them to establish credentials as peer reviewers, the already formidable problem of finding peer reviewers who represent diverse backgrounds and diverse scholarly expertise will only become more challenging.
Alison: Within HSS, there are growing questions about the shortfalls of the (double) blind process and more experimentation with changing the ways in which anonymity functions and trying to remove subjective judgments from the review process, mirroring STM developments (the Open Library of the Humanities [OLH] is but one example*). Listening to the critics of peer review, it would be easy to assume that this is a system of need of fundamental change. Yet from our experience at University of California Press, there seems to be little demand for change. No one seems ready to give up the deep level of review monographs receive, both in terms of overall assessment and deeper commentary, and I don’t see any appetite for expanding that to include post-publication commentary, for example. How do you see the system evolving and do you think more substantial change is likely any time soon?
Karin: Alison, my experience echoes yours. Not much appetite for post-publication review and mostly looking at improvements to double-blind which seems to be working. I sometimes hear about issues with a specific review/er, but usually our editors are pretty good at intervening and interpreting for authors if there is a particularly harsh review. They also let a good intellectual exchange take place — that’s the point of review, to raise the level of argument and scholarly knowledge. The AAUP guidelines address this in the sections on reviewer expectations and how to handle a problematic report; both are important for journal editors, too.
An important thing to keep in mind is that double-blind review for publication usually takes place at the end of a long train of fairly public critical exchange about a piece of work, at conferences and seminars, so there has often been opportunity for “open” review. I don’t really see “a road to reform” in that while double-blind review may not be perfect I hear often that reviewing isn’t seen as a broken system. The challenges of mutual anonymity, namely a lack of diversity among reviewers and biased reviewing, might be just as challenging in open review forums. I almost always sign my reviews, but I’m glad to have the option. I also wonder whether calls for open review are more appealing to some disciplines than others. The OLH for example, and the approach to open reviewing advocated by Kathleen Fitzpatrick, originate in literature and may best reflect movements in that field.
Mary: Like many of Alison’s fellow Scholarly Kitchen Chefs, I believe that the current system of peer review for long-form scholarly work (in whatever format it is published) is so essential to our systems of research and scholarly communication that it will only change incrementally. I suppose my work with the AAUP on this document manifests what I would most like to change: I want more participants in higher education and research to have a nuanced the understanding and appreciation of the virtues of peer review, the hard work that goes into it, the immense value it adds to scholarly communication.
*Editor’s Note: As a point of clarification, OLH has engaged in the debate around preferred peer review methodologies and questioned the status quo, in practice they offer a rigorous policy of double-blind peer-review and a clear signalling the mode of review that has been conducted on the work.
8 Thoughts on "Peer Review in the Humanities and Social Sciences: If It Ain’t Broke, Don’t Fix It?"
Something we didn’t discuss but that’s come up on Twitter yesterday and this morning is the journal/ press’s responsibility for shielding author and reviewer identity in the double-blind process. I just had a quick tutorial from our office manager about how she strips identity from the properties field in word, but also how she erases identifying information from PDFS and from track changes. I wondered if that was something that the AAUP best practices handbook might add as a minor addendum, Mary?
And among other things that exchange shows just how much at least in my field scholars do value anonymity.
I thought this was a very useful post and I would not dare to disagree with any expertise but I have one quibble. It is a post about the humanities and not about humanities AND social sciences.
All these disciplines are different one from another but as a generalisation I would suggest that in the social sciences journals are really more important than is often asserted. In spite of the fact that in economics there is a big role for reports and in anthropology for monographs and in engineering and computer science (to go into STM) conference proceedings it is articles in journals that researchers point to when we as interviewers ask about publications – see http://www.ciber-research.eu/harbingers.html for references to the context.
Anthony, thanks for your comment. I think you raise a good point, but my experience suggests more variability in the social sciences. Journals are definitely more important than in the humanities and are what matters in fields such as economics, but monographs continue to play a critical role in many fields, including sociology and anthropology. There may well be some geographic differences here and there are definitely disciplinary differences, but there continues to be a strong authorship and readership for books in the social sciences.
Anthony is right, and you can’t make generalizations even about fields broadly speaking. E.g., in philosophy (my field of graduate study in which I worked as an editor for 45 years) Anglo-American analytic philosophers rate journal articles higher than books whereas Continental philosophers rate books higher than journal articles. Similarly, in political science there is a spectrum, with American Politics at one end emphasizing journal publication and Political Theory at the other end placing more weight on book publication. In between are Comparative Politics (closer to Political Theory) and IR (closer to American Politics). Any acquisitions editor working in these fields knows these differences very well.
FAME NOT THE BEST GUIDE
One of the problems may be that Press Editors lag behind fields in their view as who is knowledgeable in a topic and so might serve as a reviewer. If topic X is under review, then reviewer Y immediately comes to mind. But reviewer Y may have made his/her reputation decades ago and demands on time that accompany fame have not permitted keeping up with the cutting-edge literature on the topic. So Y is not in a position either to review or to suggest others who might review. Of course, there are exceptions to this generalization
I would also like to clarify my reference to OLH above (thanks to Martin for pointing this out). I’m a great admirer of what OLH is doing and did not intend to suggest any negative comparison between OLH and UCP in terms of rigor or review. Rather, I wanted to highlighted the thoughtful critiques they have made of the current system and ways in which it could be improved.
Thanks for this post – very interesting and we’ll add a reference into our ‘Peer Review in Practice’ report, just released in a pre-publication version as part of the UK’s AHRC Academic Book of the Future project. We’re keen to get feedback on the report, so please do add comments onto the report http://peerreviewproject.uk/peer-review-in-practice/ Many thanks!
Readers of this discussion could be excused for thinking that double-blind review is the norm for both journal and book reviewing in HSS, but it it not. Double-blind review is very much the exception in HSS for books. Also, no mention is made here (or in the AAUP “best practices” report) of the key role that faculty editorial boards play in the peer-review process for books. Finally, reports for journal articles do not ask reviewers questions about potential markets, whereas of course that is a key question monograph reviewers are asked to answer.