Retracted research — published work that is withdrawn, removed, or otherwise invalidated from the scientific and scholarly record —is relatively rare. However, when it is inadvertently propagated within the digital scholarly record through citations, its impact can be significant, and not in a good way. Look no further than Andrew Wakefield’s notorious 1998 article, which falsely claimed that there is a link between the MMR vaccine and autism. Although subsequently retracted, it continues to be extensively cited and quoted. A number of organizations, including Retraction Watch, have worked to highlight and address this problem, and they’ve recently been joined by the Reducing the Inadvertent Spread of Retracted Science (RISRS) project, led by Jodi Schneider, Assistant Professor of Information Sciences at the University of Illinois Urbana-Champaign. In this interview, she tells us more about this work and how she hopes it will help. (Full disclosure, I participated in the RISRS project, including being interviewed and attending the workshops.)

Please can you tell us a bit about yourself — what is your current role and how did you get there?

I’m a faculty member, teaching future librarians and future data scientists. It’s been a winding path: before getting into this field I worked in insurance math, as a bookstore gift buyer, a science library staffer, a web librarian, and as a community manager for a wiki. Eventually I found my way into informatics, which led me to my current role.

What prompted your interest in retractions?

I have a long-standing interest in scholarly communication, especially how people make valid scientific arguments. I got interested in retractions as a way to think about what happens when we CAN’T rely on the results of a research paper. In particular, how does that impact later work that builds on retracted science?

The Sloan Foundation clearly agreed, as they funded RISRS. Can you tell us more about that project — who was involved, the main goals, etc?

The goal of the RISRS project is to figure out how to reduce the inadvertent citation and reuse of retracted science. Citation and use of retracted papers is common — 30 years of research, since 1989, has found this. Inadvertent use of retracted papers is problematic, particularly in clinical medicine. The main approach in RISRS has been an environment scan and stakeholder consultation, supplemented with a citation analysis.

We set out to investigate four research questions:

1. What is the actual harm associated with retracted research?
2. What are the intervention points for stopping the spread of retraction? Which gatekeepers can intervene and/or disseminate retraction status?
3. What are the classes of retracted papers? (What classes of retracted papers can be considered citable, and in what context?)
4. What are the impediments to open access dissemination of retraction statuses and retraction notices?

Overall about 70 people were involved: we had a three-part workshop with attendees from across scholarly communication as well as interviewees. I was lucky to have a diverse project team of undergrads, Master’s students, PhD students, and a part-time project manager supporting the project.

What lessons did you learn from the RISRS project? Is there anything that worked especially well? Anything you would have done differently, with hindsight?

I have a great advisory board that provided a lot of support in recruiting participants, shaping discussions, and leading ongoing dissemination. Also, recruiting people active in professional societies was particularly helpful, especially for wide dissemination of our outcomes.

Interviewing participants was a critical part of the project. It built momentum from the start and completely shifted my understanding of retraction. Snowball recruitment of participants was valuable — we were able to recruit people from roles we would not have considered if the entire list had been pre-prepared.

One limitation is that we engaged English-speaking participants, largely from North America and Western Europe. Small publishers were also not well-represented.

Few participants wanted to share position papers — it may be that we asked too late, or that that’s too far from some participants’ work. The ones we got are valuable. I keep referring back to one position paper that provides screenshots showing how one journal indicates retraction differently on three different retracted papers. Even though the work is now formally published, these images didn’t make it into the full publication.

Moving the workshop online was time-consuming. Originally we envisioned a 1.5 day in-person workshop, but the pandemic hit about six weeks into the start of the project, so we moved to three half-day workshops. Having time between the workshop sessions enabled us to update the conversation plan: we found that two weeks were better for this than one. We learned a lot about the technology along the way: sending people to set breakout rooms was easier than asking them to choose rooms; Zoom was a barrier at that point for some participants, especially in government. Technology is still a challenge for conversation-based workshops. I’d like a tool that allows a seamless move from a Zoom-style room to a spatial chat tool that facilitates mingling. The large-scale conference software that I’ve tried at recent events basically wraps around Zoom but doesn’t provide much in the way of ad hoc conversation spaces.

The pandemic’s silver lining for me has been the attention it has brought to retractions, publication quality, and their impact on policy and subsequent research. Of the many mainstream pieces this year, my favorite is the data visualization in the Economist, ” ‘Tis but a scratch: Zombie research haunts academic literature long after their supposed demise“.

Something that this project emphasized is that sharing in-progress work is really important. Before we had anything down on paper, we started planning our first presentation, at NISO Plus 2021, which had a huge impact on the uptake of the work. We shared draft reports internally a few months before we posted them publicly. In future projects I’d like to adopt a more rigorous schedule for drafts, to get even more feedback from a wider community along the way.

The RISRS final recommendations were published recently — can you summarize them for us?

The project came up with four main recommendations:

1. Develop a systematic cross-industry approach to ensure the public availability of consistent, standardized, interoperable, and timely information about retractions.
2. Recommend a taxonomy of retraction categories/classifications and corresponding retraction metadata that can be adopted by all stakeholders.
3. Develop best practices for coordinating the retraction process to enable timely, fair, unbiased outcomes.
4. Educate stakeholders about publication correction processes including retraction and about pre- and post-publication stewardship of the scholarly record.

Based on your experience of RISRS, what do you see as the main challenges in terms of retracted research?

A very small number of publications are retracted overall, so retraction can seem like an “edge case” for information systems and workflows. There’s a lot of inconsistency. It can be hard to figure out that an item is retracted, from the publisher website or from databases and search engines. In 2012, Phil Davis wrote that there should be three intervention points: before reading, before writing, and before publishing. Interventions for readers and for authors have been slow to come to fruition. Until last year CrossMark charged publishers to participate. Zotero has been at the leading edge in alerting authors and readers to retracted papers (before reading AND before writing!) — this feature has been added to other software, for instance, Papers. Tools to help journals check bibliographies against known lists of retracted papers are getting wider use. Most commonly, those are using PubMed, which is the best source in biomedicine, and possibly Crossref data. Fewer tools are licensing data from Retraction Watch, which I consider the best database for retractions, with over 30,000 items (about three times the number in PubMed).

One direct result of the recommendations is the formation of a new NISO Working Group to work on a Communication of Retractions, Removals, and Expressions of Concern (CORREC) Recommended Practice. Can you tell us a bit more about that?

CORREC will focus on how to communicate retractions. It will not address the questions of what a retraction is or why an object is retracted. Rather it focuses on what happens once something has been retracted: what metadata should be updated, how the retraction should be displayed, and how that information should be communicated.

What other outcomes do you expect (or hope!) will come out of this work?

Ultimately, if CORREC is successful, publishers, preprint servers, and data repositories will have guidance on metadata and display standards. Search engines and databases will be able to ingest consistent metadata. All of these information providers can better serve machine and human consumers with consistent, timely information, in standard formats. Consumers will have an easier time understanding what is retracted — which should reduce inadvertent use of retracted papers in follow-on work by researchers, journalists, activists, librarians, and citizen scientists of all sorts.

As a researcher, of course, I also hope and suspect that new questions will come out of this process!

What’s next for you and the RISRS project?

Disseminating the recommendations – earlier this month I spoke at COPE and ISMTE online meetings and I’m happy to speak to other industry groups. I’m revising two papers about the recommendations: a preprint under review summarizing the whole project and a more narrowly scoped article aimed at publishing professionals. And my team is synthesizing some of the literature that we collected for the Empirical Lit bibliography — currently we’re focused on a subset of 134 papers most closely related to the spread of retracted science. And of course, participating in CORREC!