Editor’s Note: In conversation with our regular Chefs, the idea was raised for us to start offering some different formats of posts, which we’ll be trialing over time. Here, the idea is to post “grab bag” articles, consisting of short pieces on a variety of subjects. There’s a lot happening in our world, and a lot to pay attention to, but much that doesn’t rise to the needs of writing a lengthy 2000-word analysis. And so we present our first “Smorgasbord” post, a buffet offering a variety of dishes. Let us know if it works below in the comments.
A Sh***y Metaphor for Science Publishing
Tim Vines
Ask most people what journals do, and they’ll probably reach for a metaphor. The one you’re most likely to hear is ‘gatekeeping’ – the idea that journals are positioned at the gate to an otherwise inaccessible (but desirable) area, and they get to decide who can enter. While it’s not inaccurate, the gatekeeper metaphor has a lot of uncomfortable baggage, and I would argue that this baggage may drive some of the current antipathy towards journals and peer review.
First, the process of gatekeeping instantly divides people into those that can enter and those that can’t. The ones making it into the compound must somehow be special or better than everyone else, leaving those on the outside resentful and suspicious that the rules for entry are unclear or rigged. It’s also much easier to identify with the cause of those outside the gate – they’ve been unfairly excluded by the likely corrupt and anonymous gatekeepers and the only true remedy is to tear down the gates and the walls.
So, while the gatekeeper metaphor does help people understand what journals and publishers do, its widespread use also fuels calls for the destruction of the journal system: anything that can be described as gatekeeping must be bad and should be brought low.
This video got me thinking about a different metaphor: journals as the sewage works standing between pipes full of untreated research and the pristine beaches of public discourse. The ‘sewage works’ metaphor brings home the importance of what we do, filtering out information that really should not become public, and improving the information that does make it out onto the beach. Tearing down the sewage works suddenly doesn’t seem like such a good idea.
Twitter, Musk, and Scholarly Communication
Rick Anderson
Given how frequently and enthusiastically scholars and scientists (especially in the humanities and social sciences) use Twitter to announce, promote, and argue about their work, I find myself watching the coverage of Elon Musk’s on-again/off-again plans for acquiring that platform with both fascination and trepidation. The invocation of free speech principles in connection with this issue has become something of a trigger – some opponents of Musk (mainly on the political left) characterize appeals to free speech as a cover for fascist ideology, while some of his defenders (mainly on the right) characterize any expression of concern over the impacts of unrestricted speech on Twitter as “radical leftism.” Neither of these postures strikes me as intellectually serious, which is unfortunate given the profound need for intellectually serious discussion around free-speech issues in the context of social media, particularly where scholarly communication is involved. Surely scientific and scholarly inquiry should be unconstrained by political considerations, and yet no reasonable person really believes that there should be no constraints whatsoever on speech. How, then, will we negotiate the tension between the need for free inquiry – a bedrock principle of academic scholarship – and the need for reasonable constraints on expression? Should some research topics be off-limits in and of themselves? Who will police the boundary between what’s debatable and what is not up for debate? I expect these questions to become both more acute and more fraught as the Musk/Twitter relationship continues along its current (i.e., as of 5 October) trajectory – assuming it does continue to do so, of course. One thing is clear: where Musk is concerned, the apparent trajectory today is not necessarily a solid predictor of what will happen tomorrow.
Accessible — Spirit vs. Letter?
Lettie Conrad
When I speak with content providers and their tech partners about accessibility, I realize our industry often addresses compliance with accessible publishing standards from a literal perspective — which has me worried that we miss out on the deeper meanings behind those standards, designed to ensure equitable access to the scholarly record. For instance, the recent OSTP guidance (aka, the Nelson Memo) directs funders to ensure “equitable access” to literature, which calls on a number of concepts implied by the word access — access as in authorization, access as in capability, and access as in comprehension.
Much of the dialogue thus far on the Nelson memo has focused on literal aspects of the first concept: access as in authorization, or the permission to read a document. Shifting the financial models from authorized access to public access is at the heart of how the open access movement has disrupted traditional publishing businesses. So, providers are working out the most sustainable methods to comply with the letter of open access mandates.
Secondarily, I’m hearing assumptions that funder mandates will expect providers to accommodate the capabilities of readers, authors, and other stakeholders. When I discuss interpretations of the Nelson Memo with colleagues, I hear most of them assume that accessible publications are those that comply with screen readers. Here again, we expect more demand to conform with the letter of the law when it comes to meeting the baseline standards of the Web Content Accessibility Guidelines or the Americans with Disabilities Act.
It’s 2022: Are Open Access Citation Advantage Studies Still not Controlling for Selection Bias?
David Crotty
I know — there are hundreds of studies like this one, and I should probably just let it go, but as a former experimental biologist and editor of the first edition of a superb book on understanding experimental design and controls, it’s still maddening every time an author adds to the pile of “evidence” for a phenomenon without accounting for obvious confounding factors. I’ve written about the alleged citation advantage for open access (OA) papers before in these pages (and elsewhere), and The Scholarly Kitchen’s Phil Davis conducted one of the very few randomized controlled trials with proper controls addressing the question. Perhaps it’s just the low-hanging fruit of what seems an easy analysis or a desire by advocates to prove a point that they really, really want to be true. I don’t know the true motivation, but the hits keep coming.
I came across this one, from a 2022 conference on Science and Technology Indicators, via Twitter: “Does Open Access Really Increase Impact? A Large-Scale Randomized Analysis“. Many open access citation advantage (OACA) studies elide the question of selection bias, which was pointed out way back in 2007 as the most likely cause for the results seen here, by referring to it as a potential confounder, then ignoring it. Here, it’s not even mentioned. To make it clear once again, comparing OA articles in a journal (or cohort of journals) to non-OA articles in that same journal (or cohort) is not a fair comparison, because the OA articles may overrepresent well-funded researchers and/or authors may save their precious OA funds and only spend them on their “most important” articles. When one truly randomizes the selection of articles made OA versus those not made OA, all such advantage disappears. And of course, given that pretty much every article is freely available through sites like ResearchGate and Sci-Hub, there is no longer any real distinction between OA and subscription articles, at least in terms of free availability.
There are a few interesting findings in this study — first, caveats aside, it seems to confirm an earlier study suggesting that publishing in fully-OA journals confers a citation disadvantage over articles in subscription journals. The authors wave this finding away suggesting that it is due to fully-OA journals being “newly-created and less attractive to high level researchers”, despite ‘journal impact” allegedly being one of the factors they did control for. Also of interest is the decline seen in the alleged OACA in hybrid journals, which I would instead suspect suggests a decline in selection bias in their samples over time. OA has become increasingly commonplace, and as more and more funders mandate it and provide funds for a broad swathe of authors to pay for article processing charges (and more and more publishers turn to transformative agreements), the differences between the OA-haves and the have-nots have lessened. The biased sample is becoming less biased over time, and the “effect” seen begins to approach the non-effect seen when the sample is truly randomized.
Discussion
16 Thoughts on "Smorgasbord: A Better Metaphor for Publishing, Twitter/Musk, Equitable Access, and Those Vexing OACA Experimental Controls"
Hmm. The sewage works metaphor seems to work best for a PLOS ONE-like peer review model, that lets everything through that reaches a certain consistency. (Forgive me.) If you want to include more selective journals in the metaphor, I think you might need a gate to the sewage works.
Anyway, I have just ordered myself a copy of the Glass book.
Good point. Maybe some sewage works also purify the water so much that they can pipe it over to the Cell/Nature/Science whiskey distillery next door?
What are the distilleries selecting for, given this study? “Authors’ names have ‘astonishing’ influence on peer reviewers: “A Nobel prizewinner is six times more likely than someone less well known to get a thumbs-up for acceptance…Palan and his colleagues invited more than 3,000 people to review [a single] paper, sometimes revealing one or other of the two authors’ names [including a Nobelist], and sometimes revealing neither…Ignorance of the Nobel laureate’s authorship boosted recommended rejection rates nearly threefold, from 23% (when the laureate’s name was revealed) to 65% (when the little-known author’s name was revealed). Advice to accept the paper outright, or with minor revisions, skyrocketed sixfold — from 10% to 59% — when Smith’s authorship was declared.” https://www.nature.com/articles/d41586-022-03256-9
Since you asked for feedback on how the “smorgasbord” concept works, just a quick and possibly picky comment: I think it would work better with a little more attention to some low-hanging fruit in formatting. Specifically, this might include a bit more (and consistent) space/separation between the individual stories; space between paragraphs in all stories (not just the last one); and shorter paragraphs in some cases, such as Rick’s contribution (and possibly David’s).
My experience has been that these kinds of things, while seemingly minor, make articles a bit more accessible in online contexts, and it seems to me they would improve the “smorgasbord” offering. (Sorry if this seems a bit too focused on form rather than content.)
That one is on me.
WordPress’s WYSIWYG interface is not great for subtle differences and each of the things you mention appear identical in the back end tool but different in the published post. Our posts come in from a lot of different places (email, Word Docs, Google Docs, etc.) each with their own idiosyncrasies. The only way to really clear up these issues is to dive into the html, and some days there’s time, some days there isn’t. But I’ve done my best to clear these up, thanks for bringing them to my attention.
Many many years ago I sat in on a lecture by a Harvard history professor that seemed to sum up the free speech question quite nicely. He was talking about Lincoln, emancipation, and reconstruction when he channeled Karl Popper and asked:
If one lives in a truly tolerant society, doesn’t it follow that such a society must necessarily be intolerant of intolerance?
I like this smorgasbord format very much. It is very thought provoking and a great conversation starter. It would be great if there were links at the end of each entry to a more meaty continuation or expansion of the entry for those who wanted to read more (although this might defeat the purpose of this article type). The short pieces are perfect for getting the flavor of the topic in a few minutes and then being able to come back and read the next entry at a different time – which is so beneficial in a work/life environment that is prone to frequent online or in-person interruptions. Overall, I am looking forward to reading more of these installments.
Hi Becky – thanks for this comment. I can’t speak for the other authors on here, but one of the reasons I like these Smorgasbord pieces is that I can put together a few paragraphs on a topic without having to agonize over how to expand it into a (readable & useful) 2000 word post. So, sadly, there won’t be a full length treatise on the sewage works metaphor for academic publishing anytime soon…
Re: Metaphors for Publishing
The metaphor of “gatekeeping” in the scientific publication process goes back to the early 1960’s. Described by sociologists of science, it was used to describe the formal structures that maintained the strata of academics: who was allowed to join the elite colleges and universities, and who was allowed to circulate their findings among their community. Nowhere could I find reference to disseminating science to the public. Gatekeeping was used in the same way country clubs boards deciding who was allowed to join.
Crane D. 1967. The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. American Sociologist 2: 195-201. https://www.jstor.org/stable/27701277
Hi Phil, thanks for adding this reference to the mix. I’m not quite clear what you mean by “Nowhere could I find reference to disseminating science to the public” – the Crane piece contains the line “De Grazia has suggested that journal editors are the ‘gatekeepers’ of science”, and throughout the first page (which is the only part I can access) she consistently brings up the idea that authors from prestigious universities have an easier time getting published because editors are biased in their favour.
In the sociology of science (for what I can remember from grad school), science communication was largely focused on communication *among* scientists and not the transfer and translation of science to the public. Gatekeeping was about *who* got their papers into the top journals, most of which did not practice double-blind review. So, there was skepticism that editors would only allow authors like them into these journals, set the agenda for research, regulate who speaks for the community, and ultimately establish the next in line for journal editorship. Not surprisingly, the notion of meritocracy in science was being questioned.
Research [1] in which manuscripts were stripped of prestigious authors’ names and elite affiliations, replaced by unknown entities, and submitted back to the journals in which published them attempted to reveal just how much editors and reviewers were influenced by social strata in academia. This paper set up a huge controversy in sociology and immense negative blowback to the authors and their careers. It even spawned a conference focused entirely on this paper.
[1] Ceci SJ, Peters DP. 1982. Peer Review: A Study of Reliability. Change 14: 44-8. https://doi.org/10.1080/00091383.1982.10569910
So “resentful and suspicious” researchers who “[call] for the destruction of the journal system” because of its gatekeeping function are going to be mollified by … having their output likened to raw sewage?
More seriously, yes, academia has its share of critics of peer review as currently practiced, of the power of “prestige” journals, and so on. But I suspect that the majority would be content to maintain the current system of peer review and gatekeeping. After all, it is academics, in their roles as peer reviewers and editors, who are the decision makers that sustain the current structures. Journals are only the vehicles for this activity.
Scholars don’t seek the destruction of the journal system because they are ignorant of filtering, or object to gatekeeping.The real objections to the journal system are, economic ones, namely that the large publishers, through their control of legacy systems and historically valuable titles, are able to extract huge rents from the scholarly community. Lest we forget why we are angry, publishers helpfully provide reminders multiple times per day by popping up that window asking $49.99 for a peek at an article that might or might not be helpful. It surprises me that you’ve never heard these issues raised before. I can’t imagine what would lead you to think that academics want to bring down the system so that their work can evade scrutiny, or that they need some obscene metaphor to teach them the value of filtering.
Thanks for this comment. Working as the managing editor of a journal/sewage works was what led me to think that some academics would prefer to evade scrutiny and just get their article out there regardless of its quality.
In fact, to torture my metaphor a little more, the confidential nature of the peer review process means that researchers are generally only aware of what goes down their own or their colleagues’ pipes, and what emerges onto the beach (i.e. is published in journals). They’re blissfully unaware of what else the sewage works is dealing with, and in what volume…
I like the outcome of the smorgasbord format, which is that I have a bit more knowledge and updates on an array of topics, but it reminds me a bit of an email newsletter that could/would fulfill that value proposition nicely. Currently, the email I get from SK highlights a single post with perhaps a teaser of another resource or two. Information hoarders such as myself will often clip an article into a relevant notebook or tagged area for later use and reference, so that is made somewhat harder in this approach. Just my two cents on usability FWIW, and kudos for trying things out.
I am a fan of the smorgasbord format!
I like the format as well. While not a perfect comparison, it reminded me in a good way of the Brief. I always appreciate links to articles/content I missed or that are of interest, so would appreciate authors of future Smorg pieces keeping that in mind.
