Editor’s Note: Today’s post is from Sara Rouhi. Sara manages business development for Altmetric in the US and Canada. She is an active member of the Society for Scholarly Publishing’s Education Committee and the 2015 recipient of their SSP Emerging Leader Award. We’re cross-posting today with the Altmetric blog.
This past week, three distinct moments came together for me prompting me to write this blog post: a peer review conference, a debate on the journal impact factor (JIF), and an art exhibit.
I’m just coming back the 8th International Peer Review Congress (PRC) in Chicago. This once-every-four-years meeting brings together journal editors, authors and anyone else working in the journal editorial or production space to discuss the “big questions” facing the peer-review process.
While in Chicago, I visited the Museum of Contemporary Art (MCA) – one of my favorite US art museums – to explore their Takashi Murakami retrospective and stumbled upon the “Politics and Power Plays” exhibit featuring artists examining the intersection of politics and power.
Several weeks prior to this trip, I had the privilege to participate in the Charleston Library Conference’s famous “Hyde Park Debate.” This Oxford-style debate (complete with prepared opening statements and first rebuttals) posits a statement and asks experts to argue for and against it. This year the debate statement for support/rebuttal was “Resolved: The Impact Factor does more harm than good.” (Which do you think I argued?)
What do a journal conference, art exhibit, and webcast debate have in common?
The “Politics and Power Plays” exhibit and the PRC meeting/Charleston debate examined similar themes at the macro and micro levels respectively.
The MCA exhibit used art as form to highlight how structural inequities in power, access, and opportunity are foundational to liberal democracies and may ultimately render them unstable. The PRC and Charleston debate – with their examination of the inherent biases in the western research paradigm – emphasized how the barriers to entry in this space are silencing voices whose research is vital to the research endeavor.
Charleston Hyde Park Debate: Beyond the Usual Arguments
When I initially accepted this exciting challenge, I planned on taking the standard approach and arguing the “go-to” list of Impact Factor “against” positions everyone is familiar with. When I really stopped to reflect on the topic, my inner political scientist rose up, and I realized that despite being one of the few employees at Altmetric/Digital Science with a social science background, my expertise in sociological and political theory might finally be brought to bear on the thorny question of metrics in the research paradigm.
My undergraduate and graduate degrees were both in political theory and the study of how culture impacts the formation of definitions of self and community (and by extension, politics). Evaluating this year’s Hyde Park debate statement through the lens of critical and post-colonial theory, I realized there was a lot more to probe about the journal metrics than just their pros and cons as evaluative tools.
I decided on more a radical approach toward my argument and focused on the structural inequities perpetuated by journal based metrics, rather than focusing on the same arguments everyone has heard. My hypothesis/argument was simple: the group in power makes the rules and the rules typically favor the group in power.
In this case, I coined the term “western research industrial complex” (because who doesn’t love adding ‘industrial complex’ to anything they’re arguing against?) to describe the dominant power paradigm of the global research landscape: white, western, English-speaking men and the prestige institutions they represent.
These folks created the paradigm, its rules of engagement (Study with this professor! Postdoc at that institution! Publish or perish!), its costs for entry (JIF! Citations!), and its prizes (Tenure! Grant money! Status!). Everyone else – the global south and non-white, non-western countries – starts at a disadvantage by not belonging to the dominant group and constantly confronting the structural inequities of that exclusion.
The Peers Who Review Me: White, Western, Men
The PRC meeting further examined these questions with almost half the presentations addressing bias in the peer-review process: a bias that generally excludes those outside the “research industrial complex.”
The big question facing authors, editors, and publishers at this meeting was the question of whether and how to tackle the inherent biases built into the peer review paradigm. This paradigm is dominated by white, western, English-speaking men who have come of age in the “research industrial complex” and benefit greatly from their association with it (see my comments on Eugene Garfield and his development and monetization of journal metrics in the Hyde Park Debate opening statement here.)
While this group likely does not intend to allow bias into the peer review process, the data examining the outcomes of this process tell a different story. Research going as far back as 2013, confirms that for various structural reasons, women receive fewer citations than men. The journal selection, peer review, and acceptance process is a critical part of this. (See Alice Meadow’s report on PRC from last week’s Scholarly Kitchen.)
When speaking to the powers-that-be in western publishing, the chorus is often the same. “We’re scientists. We’re objective. We’re not biased in our assessments. If the research is good, it will get published.”
Alas, as much of the research at PRC proved, that is not the case. While these individuals genuinely believe they’re objective, the idea of a truly “objective” human being is laughable. Max Weber, Emile Durkheim – indeed any student of sociology would snort in incredulity at the idea that a lifetime of experiences does not color judgements about the world.
It’s the definition of privilege to make this naïve assertion. Ask any woman who has mulled over what to wear to a job interview or any black person who has contemplated expressing an unpopular opinion in the workplace. Can’t identify with those examples?
Consider the Latina who honestly admitted that her background as a “NewYorkrican” would impact her perspective as a judge on the Supreme Court, Sonia Sotomayor. Essayist and public intellectual, James Baldwin, writing during Jim Crow put it another way:
For it means something to be a Negro, after all, as it means something to have been born in Ireland or in China, to live where one sees space and sky or to live where one sees nothing but rubble or nothing but high buildings, We cannot escape our origins, however hard we try, those origins which contain the key – could we but find it – to all that we later become. [emphasis mine]
We bring our experiences, exposures, and education (or lack thereof) to everything we do and the presumption of objectivity does the greatest disservice to those most effected by bias.
And none of this is new.
Playing Politics: The Quest to Maintain Traditional Hierarchies
Indeed, theorists of power (most famously Friedrich Nietzsche, Michel Foucault, and Hannah Arendt among others) have been top of mind for me lately as they all highlight the same underlying truth: Systems of power insinuate themselves so subtly that their captives not only accept but embrace the norms and conventions of the system even if those norms disadvantage them.
The “Politics and Power Plays” exhibit speaks directly to this insinuation. The opening statement of the exhibit reads:
Power profoundly affects our lives yet often operates invisibly; consequently it’s easy to take it for granted. This selection of art drawn from the MCA Collection considers the ways that artist give power a form in order to name it and work with or against it… All of the works look critically at power structures from the position of the artist, who stands outside traditional hierarchies. [emphasis mine]
The challenges seen in the US – whether it’s a question of taking down Confederate statues or allowing transgender citizens to serve in the military – are more apparent than they have been since the civil rights movement of the 1960s.
This past election revealed that many of our neighbors, friends, work colleagues, and family members value traditional hierarchies and maintaining them. While the American political experiment was built on the promise of “equality under the law” and “life, liberty, and the pursuit of happiness,” it’s easy to forget that those inspirational documents were forged under an economic system reliant on chattel slavery, where each black person counted as only 3/5 a person.
This past election did us the painful service of momentarily revealing the power matrices that ceaselessly entangle the many while benefiting the few. Can I fly back to the US from a visit to Iranian relatives in Canada while traveling on a green card? Will my cleaner, an immigrant with no papers, be deported despite having American children, owning her own business, and paying taxes for decades in the US? Will my recently married gay friends still be recognized as married by the state? Will my elderly black neighbor still be able to vote when her drivers license has expired and she has no access to the internet?
Before we can address the impact of traditional hierarchies on those at the bottom and top of the food chain, we must observe that they exist and unflinchingly acknowledge where we sit within that hierarchy – journal editors and engaged citizens alike.
As a woman, an immigrant, the child of a Muslim, and a person with brown skin – how long will being a US citizen protect me?
These are real questions I have asked myself in the last 7 months since my friends, colleagues, family, and fellow citizens elected an ethno-nationalist president.
So all this to say…
Before we can address the impact of traditional hierarchies on those at the bottom and top of the food chain, we must observe that they exist and unflinchingly acknowledge where we sit within that hierarchy – journal editors and engaged citizens alike.
Academic publishing and the research industrial complex particularly need this kind of self-reflection to acknowledge their role in vetting research and passing judgement on what is included vs. excluded. We don’t need to read the latest literature on peer review bias or to (re)read Discipline and Punish to see the effects “politics and power plays.”
It’s incumbent upon all of us – trapped within the myriad hierarchies that we inhabit – to shed light on them and call out inequities as we confront them, whether from a privileged position or not. (My argument for the Charleston Hyde Park debate was my small attempt at doing this.)
As I fly back from Chicago relieved that my “Premier Access” status allows me to skip the line and check a bag with no fee, I find I’m running into (and benefiting from) structural inequities at every turn. 90% of the battle is recognizing them and the last critical 10% is deciding which side you’re on.
So whether you’re pro- or anti-JIF, or pro- or anti-Brexit/Trump, remember that these inequities are all around us. Notice them, check your privilege, and do your homework.
This blog post does not reflect the position of Altmetric or its employees and is solely my position as a member of the resistance.
Discussion
22 Thoughts on "Guest Post — When Metrics and Politics Collide: Reflections on Peer Review, the JIF and Our Current Political Moment"
Forgive me for being obtuse, but what on earth are you trying to say?
I thought these two points were a pretty good articulation of what Sara is saying:
“We bring our experiences, exposures, and education (or lack thereof) to everything we do and the presumption of objectivity does the greatest disservice to those most affected by bias.”
“Academic publishing and the research industrial complex particularly need this kind of self-reflection to acknowledge their role in vetting research and passing judgement on what is included vs. excluded.”
If this argument is true, and I don’t believe that it is, then there is no point in making any argument, as we are all imprisoned in our social categories. Thus someone from my category cannot influence someone from a different category. It’s truly a depressing way to view the world.
Joe, my takeaway was more that we are all *potentially* “imprisoned in our social categories” but that by being conscious of that risk we can mitigate it.
Sara’s post doesn’t come as a shock, even to a gray-haired white male like me. But it’s certainly bracing to see it on my computer screen first thing on a Monday morning.
If we’re smart, as individuals and companies we need to be a bit introspective about who we are and how we got there. I don’t think this means the “western research-industrial complex” developed with malign intent. Like all social and economic systems, it grew more or less organically. But now that we’re seeing that the system contributes to patterns that are at least out of sync with the contemporary world, those of us who have benefited from the system should at least be open to different ways of looking at things.
Since I grew up on Thomas Kuhn’s book “The Structure of Scientific Revolutions,” I’ll fall back on his familiar formulation: The “western research-industrial complex” doesn’t work for emerging kinds of scholarly activity or for many emerging groups of scholars, so it’s time for a new paradigm. In due course, the paradigm probably will emerge from relative newcomers to the field. The aim for those of us who have benefited for decades from the current paradigm is not to feel guilty about what we wrought, but to be open to the need for something new.
I’ve been a referee on manuscripts for many years and not once have I detected whether or not the unknown author was white, Western or male. Even if the author’s first language seems not fluent English, the decision was solely on the merit of the research question and data analysis, regardless of method or paradigm. Seems like a convenient excuse for rejection.
Perhaps you skimmed over this section of the post above:
When speaking to the powers-that-be in western publishing, the chorus is often the same. “We’re scientists. We’re objective. We’re not biased in our assessments. If the research is good, it will get published.”
Alas, as much of the research at PRC proved, that is not the case. While these individuals genuinely believe they’re objective, the idea of a truly “objective” human being is laughable. Max Weber, Emile Durkheim – indeed any student of sociology would snort in incredulity at the idea that a lifetime of experiences does not color judgements about the world.
It’s the definition of privilege to make this naïve assertion. Ask any woman who has mulled over what to wear to a job interview or any black person who has contemplated expressing an unpopular opinion in the workplace. Can’t identify with those examples?
Consider the Latina who honestly admitted that her background as a “NewYorkrican” would impact her perspective as a judge on the Supreme Court, Sonia Sotomayor. Essayist and public intellectual, James Baldwin, writing during Jim Crow put it another way:
For it means something to be a Negro, after all, as it means something to have been born in Ireland or in China, to live where one sees space and sky or to live where one sees nothing but rubble or nothing but high buildings, We cannot escape our origins, however hard we try, those origins which contain the key – could we but find it – to all that we later become. [emphasis mine]
We bring our experiences, exposures, and education (or lack thereof) to everything we do and the presumption of objectivity does the greatest disservice to those most effected by bias.
A point to consider is that, if from a typically underrepresented group, the author DID consider their (ethnicity, race, gender, sexuality, SES, etc).
As a Humanities PhD, I also have no difficulty in accepting the premise of this column. I would suggest reading the great literary masterpieces of the world to get some sense of how belief systems change across space and time. In my opinion, the major function of literary criticism is to tease out the differences for those who are reading the works, perhaps incorrectly, from a different world view.
As for the scholarly communication system being objective, I would suggest trying to succeed in another scholarly culture that most likely considers its system to also be objective. I would expect that the rules would be set up to favor those who were successful in this system and that the “new” scholar would profit greatly from examining what these rules are.
The big potential take away from this column is to have an improved ability to step back and to recognize that any “objective” system has assumptions that are probably not provable but that undergird how the system works. I have had hard core qualitative researchers argue that literary criticism isn’t research because it doesn’t deal with data, but Shakespeare may tell us more about the world than a thousand scholarly articles.
Finally, I recognize my limits. I understand the world I live in very well and believe myself successful, but I wouldn’t last very long on the streets of the blighted areas of Detroit unless I took a crash course in learning new assumptions and how they should affect my behavior.
So whether you’re pro- or anti-JIF, or pro- or anti-Brexit/Trump, remember that these inequities are all around us. Notice them, check your privilege, and do your homework.
I don’t think anyone in publishing disputes this claim. The industrial-publishing-complex is aware of systemic biases (note all of the research conducted on this topic) and has been pretty open to experimenting with new ways to attenuate power inequalities. Peer review is a relatively new (post war) invention. Blinding the peer review process is even newer.
Throwing up one’s hands and not willing to engage in very practical issues, like taking a side on whether the JIF does more harm than good, is neither helpful nor constructive here. Unless you really want to keep the old white guys in power, you’ll have to offer up some alternative. Perfect is the enemy of the good.
To be fair, the author of this post did indeed take a position on the JIF, and argued it does more harm than good:
https://www.charlestonlibraryconference.com/wp-content/uploads/2017/08/Opening-Statement-Sara-Rouhi.pdf
I’m not sure the post’s point is that we should all just give up, more that we need to understand the biases inherent in the system in order to improve it. It also offers a cogent answer to a question I regularly hear — we’ve been complaining about the JIF for decades now, yet it remains entrenched, why is that? As the author states, those with the power to change the criteria academia uses for advancement are in that position because the current system worked really well for them, which is likely why there’s been so little movement.
Even if we acknowledge the shortcomings of the JIF, it works in certain circumstances for particular purposes. The usual example from physics: Revolutionaries like Einstein and Schroedinger showed that scientists’ understanding of physics was fundamentally flawed. However, good old Newtonian physics works perfectly for things like calculating satellite trajectories or designing bridges. The new and the old coexist. It’s not realistic to think of something completely extirpating JIF (or altmetrics), but it is reasonable to hope for a new analytical approach that will work when the established systems break down.
With ed boards being more international in scope I wonder if the data supports the argument.
Perhaps it helps to provide one form of diversity, but it may not be helping to address issues of gender or class / social spheres.
The problem transcends the evaluation of the journal/articles. There is a proclivity to look for simple metrics in making decisions such as the value of a journal for subscription services, the use for promotion/tenure, and awards of grants and other recognition. In other words JIF’s are metadata and often default measures. Similar to Phil’s comments, other “standards” get accepted as default, such as double blind reviews. All are, like force fitting Cinderella’s slipper on the sisters, taken across disciplines from the physical sciences to the humanities as other means of ranking enter like seals of quality on a bottle of organic catsup.
The “cultural” issues of the JIF extend to the culture that accepts this and other metadata as a stamp of approval.
I also accept the premise and took it that the JIF does more harm than good. Applying it as a way to judge someone’s science, e.g. for tenure, can be replaced with what formerly happened. Those judging evaluated the science (study) or theoretical article itself, trusting their expertise to make the judgement, and not basing judgement on where it was published. Academics are giving away their peer reviewing in this case, instead deferring to a number. To look at Kuhn does make sense. The idea that knowledge development changes through a revolution means that the status quo, in this case, a metrics driven system, must be disrupted. That cannot be done by one person. The innovative, or counter-paradigmatic work that may be “ahead of its time”– a revolution, may well only appear in a low IF journal. I do not think it reasonable to conclude anything based on the IF. In addition I think we should not let the more organic ways of evaluating new knowledge be supplanted by the business model and constant use of metrics.
Joanne’s comments reflect my earlier comments. The problem with the use of metadata, including IF in the publishing arena relates to an earlier discussion which raised the issue of the continued expansion of journals and the demand for pub/perish by authors where one might consider these “proliferation wars” and whether there was a need for slower, more deliberate publishing of research.
Can the academic community, including the journal publishers accept Joanne’s “more organic ways of evaluating..” when there is pressure from all parties for rapid publication/evaluation of scholarly production. Perhaps such a path might allow a reduction in practices that are problematic. Or is the arena headed on a spiral like Cassini?
It is my current major issue. Read the work when evaluating, don’t just look at the IF. If authors could publish again in the most appropriate or the most socially just venues for their work, without having to worry about IFs because they knew their work would be fairly evaluated, this could reduce the stranglehold of the big 5 publishers as they continue to buy up everything.
Great post, Sara. (That’s all!)
Thanks, Charlie. Delighted to get a conversation started here. Thanks to TSK and David for making space for it!
Thanks Sara for a really thought-provoking post on an important topic. Although we (should) all know that institutional bias exists that’s not necessarily the same as recognizing what that means for us as – in particular, privileged – individuals. And meaningful change won’t happen until we do really acknowledge that, both as individuals and as a community.
I am impressed by what a good writer Sara is, especially when she’s writing from the heart. The article is so moving, in fact, that it took a few hours for some concerns to make themselves known.
One concern is that viewing the problems of academia, such as inappropriate use of the JIF, through the lens of identity politics seems to focus on our differences precisely when we need to unite against a common enemy. Science faces huge problems, such as this administration and its hostility to science, that require a stable and strong coalition to fight. We’ve seen already the adeptness with which they use scandal and drama to create division among us, while they remain united and focused on their regressive aims. Science needs all of us working together to defend it.
The second point is, I think, the more serious issue. Science plays a unique role in our democracy and in any society which aims for egalitarianism. The truths it uncovers can be found by anyone, even a lowly patent clerk or Franciscan monk, and recognized by all. Perhaps those of us in the bubble of STEM research don’t appreciate how unique this is. In every other domain, truth comes from esteemed authority figures. In science, it comes from measurement, observation, and analysis. We, uniquely, do think we can discover objective truth. There’s no question that we see the world through the lens of our experience, as the great work by Project Implicit and others have shown, but we do uncover objective truths. If you drop an object in a vacuum, it’s going to accelerate downwards at the same speed, 9.8 m/s^2, no matter who is observing it. The essence of the scientific method is removing bias, but understanding it can never be removed entirely & thus must be controlled for. If we say there’s no objective way to assess the value of research, don’t we open ourselves up to a yawning chasm of false equivalence in everything? How much difference is there between “You can’t review my grant fairly because you’re a white man and I’m a black woman” and “You can’t review my grant fairly because you’re an atheist/pro-vaccine/believe in global warming.” So, Sara, I hope I’ve read you wrong here and that you do believe that we can come up with ways of deciding which research findings are true that are independent of the assessor. If not, well, I would say you’ve correctly diagnosed the patient – assessment of research contains bias – but your treatment will probably kill the patient.
Lamar Smith et al. are salivating at the chance to knock research off the pedestal and make it submit to political whim. If we can’t unite & double down on our ability to be objective, we lose our only really unique advantage and defense against his aims.
You’re brave for speaking up & did so eloquently, but if you do so not at the disadvantage of white men, but everyone – patients, business owners, farmers, and up many others who depend on the truths uncovered by research.
