Editor’s note: Today’s guest post is by Gráinne McNamara, Research Integrity/Publication Ethics Manager at Karger Publishers and COPE Council Member; Jeremy Ng, Editor-in-Chief, Journal of Complementary and Integrative Medicine, Scientist at the University Hospital Tübingen Stuttgart, Germany and Assistant Professor at McMaster University in Canada; Elizabeth Moylan, Senior Manager, Research Integrity, Strategy & Policy, Wiley; Coco Nijhoff, Senior Teaching Fellow, Library Services, Imperial College London and and Lauren Flintoft, Research Integrity Manager, IOP Publishing.

Last month, the Committee on Publication Ethics (COPE) hosted its third Publication Integrity Week, featuring a week-long series of online events bringing together a diverse group of speakers, practitioners, and community members to examine pressing challenges and opportunities in scholarly publishing. As part of the week, Gráinne McNamara hosted a panel discussion to continue the conversation around AI (artificial intelligence) use: policies and reality — with a range of perspectives shared by Jeremy Ng, Elizabeth Moylan, Coco Nijhoff, and Lauren Flintoft.

In this session, 84% of the surveyed attendees said they are using AI “regularly” or “daily” in their work, reflecting the reality that generative AI (GenAI) is here, presumably to stay. However, it can seem that users and those creating policies are running to catch up with where the capabilities of the technology are. In this context, we considered what we can realistically expect from our researchers, authors, reviewers, editors, and readers, and how we can ensure our policies and activities are pragmatic and adhere to the principles of research integrity, now and in the future. Here we share our reflections from the discussion.

Jeremy Ng

One of the things that’s really fascinated me over the past couple of years is what the different stakeholders in academia who use AI really think about AI, how they perceive it, and their attitudes towards it. We’ve surveyed editors, researchers, students, peer reviewers, and I think one of the things that has become quite clear to me is that many people, across these different stakeholder populations, are unaware of, or potentially misunderstand, existing AI policies.

It’s impossible to truly know whether researchers have used GenAI. However, as an editor, what I’ve observed is that the vast majority of authors are submitting manuscripts that don’t declare AI usage and don’t tick the box to confirm this, even though it has been estimated that many researchers are likely using GenAI. I think this hesitancy is due to misconceptions that if they say that they used AI, then that will immediately disqualify their manuscript, even though, in my role as editor, that is not what I would plan to do. I also see that students are often hesitant to talk to me about AI even though I’m more interested in knowing whether and how they used AI or not, not in a punitive sense, but rather, to have an open and transparent conversation about it. Openness could lead to further discussion about the best ways to use it, and what may be less appropriate ways to use it. It’s challenging to create pragmatic policies without openness.

The AI world is changing so rapidly that there needs to be the establishment of some sort of mechanisms for regular policy updates that are informed by updated stakeholder feedback, by technological advances, by empirical evidence of AI use, and by new research that is emerging. People also admit that they are interested in training, and they want training, and yet they don’t necessarily even know where or how to access that training. From my conversations with faculty members at different institutions around the world, one of the biggest challenges at the institutional level is that there’s just no standardization yet in terms of how to train students around acceptable and unacceptable uses of GenAI. There also needs to be more awareness about what is appropriate versus inappropriate use of AI, and more consistent guidance from journals and publishers so that researchers can keep up with current policies.

I think it is important to participate in events that are open discussions, as we had at COPE’s Publication Integrity Week. COPE has the potential, with publisher membership and institutional membership, to bring people together who have different perspectives and lived experiences around AI use and the potential pitfalls and challenges. Sharing those experiences and knowledge can really strengthen capacity building, so we can conduct more research on AI and learn about people’s interactions with it. There is a general need for greater collaboration and more open dialogue between different stakeholders. From what I’m seeing through surveys is that trainees and students are looking to their advisors for guidance, and their advisors, who are faculty members and who are researchers, are looking to publishers and journal editors for guidance, and so I think we need to have more collaborative discussions.

We all have a responsibility to lead by example. We are living in a world where we’re interacting with AI, and if we choose to use it for a certain portion of our work, where we feel that it is justifiable and ethically sound to do so, then we should take the lead and disclose this.

Elizabeth Moylan

We all have a role to play in supporting our communities when it comes to the responsible use of GenAI. What we’ve been trying to do at Wiley with our ExplanAItions study is listen to what researchers need in terms of guidance on the use of tools, and how usage should be declared, and we provide information and training to support those needs. Through our surveys with the communities we work with as authors, reviewers, or editors, we have developed guidance for researchers, which is continuously iterated upon as feedback and experiences grow.

Of course, the work doesn’t just stop there. We must make the information practical, cover the legal aspects of using GenAI tools, address people’s needs, and provide clear guidance and policy. That in turn throws up more questions, and so we’re continuing that work. The usage of GenAI has really leapt in the past two years; it’s moving so quickly, and that’s part of the challenge. One practical aspect of that is tackling the perceived stigma of disclosure and providing ways of addressing that. We all need guidance and training to help, and we really want to support positive reinforcement. We are also planning more tailored support for each community with FAQs to make the guidance more accessible while recognizing that approaches are going to change and adapt. So, we’re not standing still.

We are all learning and sharing in this space, and continuing that open dialogue will be key to discovering what works well, what doesn’t work so well, and enabling us to critique approaches to a particular use of an AI tool so that we can all collectively build on that. To echo many of the conversations during COPE Publication Integrity Week, true progress comes from collaboration and working together across all sectors, institutions, journals, and publishers.

Coco Nijhoff

Policy around the use of GenAI cannot be developed in a vacuum. When we look at the actual practices of researchers, particularly as authors, what we have seen from a recent survey in Nature is that people are very willing to use AI; however, people are afraid to disclose their use. It’s interesting to explore that and unpack it further as a community. How can we work toward a culture where we have a better understanding of the things that GenAI will do throughout this whole process — not only with respect to writing, but in other activities, for example, generating ideas, outlining, mapping, and developing a research angle. Of course, we need to ensure that all the different uses of AI are open and transparent, and explore what is okay and what is not okay. We need to work together towards a more open culture of trust.

Each of us has a role in building this culture of trust and integrity, and each of us has different opportunities within our roles. Not only are we individuals, but we’re also in teams, we’re part of a department, within an institution, and that shapes every interaction we have. At Imperial College London, we are seeing postgraduate research students struggling with the same issue around disclosure. They’re using GenAI, but they are uncertain how much they can, or should, disclose about GenAI use. Capturing those concerns, observations, and evidence could inform the development of policy. We’re all involved in developing policies to support as a community-based endeavor, so using that evidence to inform policy is something we want to explore further.

Following on from policy is practice, and incentivizing or motivating adherence to best practice. Policy can sound very firm and very formal, because the institution or the publisher has a role to play in upholding standards and academic integrity, including research integrity. If we explore the relationship between policy and guidelines, could there be a way to marry the two together to be supportive and helpful? Beyond simply explaining the need for disclosure, is there a way to demonstrate or share the benefits of disclosing and being transparent around that?

This, in turn, will enable more discussion about what we can consider to be acceptable use and how to articulate what that looks like, and if it’s even possible to reach consensus across disciplines, institutions, or even countries. Having conversations about the skills and competencies and range of uses for AI enables dialogue, and that ongoing conversation is critical to address the challenges. In this way, we are anchoring the fact that AI exists, from this point onwards, and we’ll see where we get to in five years’ time.

Lauren Flintoft

We need to ensure that we support our communities by listening to them and getting that feedback so that the right balance is struck with regard to GenAI policies. Otherwise, we potentially run the risk of implementing a policy that is not fit for purpose.

While there has been a lot of focus on how authors use AI, reviewers play an important role. Reflecting IOPP’s broader study sharing Insights from the global reviewer community, one thing we’re seeing is reviewers using GenAI tools to support not just improving the language and drafting of the review, but also using it to assess the manuscript. This goes beyond many publishers’ policies, including IOP Publishing Policy. A point that I continue to emphasize, although this may perhaps change in the future, is that AI should be used as a tool. If AI is used to replace core responsibilities of any stakeholder’s role, or used beyond simply a function of a tool, this is where potentially irresponsible use could occur, for example, quality issues being introduced into peer reviews and manuscripts. The individual is “the peer” in peer review, and critical thinking is the foundation of the peer review process. If a reviewer uploads a manuscript to an AI tool and asks for a peer review report, in addition to copyright and confidentiality concerns, the added value for the authors is effectively zero because they are seeking critical feedback from the individual themselves. When we see reviewer reports that are somewhat shallow in the assessment of the research, with vague feedback and mistakes such as hallucinated references, these are all potential flags that GenAI has been used. This, of course, leads to a breakdown of trust in the peer review process.  Maintaining trust forms the foundation of our policies and many publishers’ policies.

It’s understandable that researchers as authors and reviewers look to publishers, their institutions, and organizations to understand the responsible practices and expectations around AI use. Something that we do at IOPP is to offer our reviewers a peer review training program. Through this, we can explain what the responsible uses of AI are and provide more educational and practical support so that they’re aware of publishers’ expectations and how they can responsibly use GenAI.

Gráinne McNamara

A key takeaway is the need to have open discussions with our communities and understand how they’re using AI, and what they need in order to develop pragmatic policies that are fit for purpose. There’s a clear case for iterative and informed policy setting that can evolve and keep pace with the spread of AI use, and institutions and journals offer an important point of contact when people need help and support. However, we must be aware of the risk of having a “tail-wagging-the-dog” situation where policy follows practice instead of the more considered approach of determining what is appropriate practice, then setting appropriate policy that then further informs practice. Reflecting on what our end goal is when we set a policy, as well as how to incentivize best practices, came out as a strong throughline of this discussion. Of course, cultural and behavioral change doesn’t happen overnight, but we are all on this journey together.