At the start of every December, The STM Association hosts their innovation and integrity days in London. Over the past few years, this event has morphed from what used to be known as ‘STM Week’, a series of three, traditional one-day conferences, to a unique two-day program consisting of an innovation fair and the Integrity Day.

The fair consists primarily of a series of lightning talks with booths for startups and innovators to meet collaborators and potential customers, and also hosts the Vesalius innovation awards. The Vesalius awards — of which I’m proud to say, I’m a judge — is run by Karger and was co-hosted for the first time this year by the BMJ Group. Now five years old, these awards combine a recognition for outstanding innovation in medical publishing with a program of mentorship. This year, congratulations go to Knowledge Gate Group whose product puts life-science companies in touch with experts in academia to foster collaboration and engagement. Runners up were Prof. Valmed, which provides validated information to clinicians supported by retrieval augmented generation (RAG) and Clear Skies — more about them later.

The Integrity Day has a different tone. Conducted under Chatham House rule to encourage frank and open discussions, the one day seminar explores difficult topics around research fraud, paper mills, and steps that might be taken to protect the integrity of the scholarly record. This year, STM called the Integrity Day ‘Trust beyond the manuscript’, with researcher identity and image fraud being big topics, not surprisingly as STM has recently released reports on both of these subjects. In the interests of transparency, I should point out that I wrote the image fraud report.

One of the things that makes STM Week special in my opinion, is its proximity to the holiday season, so it has become a combination of a yearly recap and an annual festive get-together, where industry leaders, entrepreneurs, and innovators get a chance to swap ideas and try to invent the future over coffee breaks, receptions, and dinners. In other words, it’s a good way to find out what’s on the industry’s collective mind. This year, research integrity loomed large over the whole week. In the two years since the STM Integrity Day started, many publishers have invested in research integrity offices that investigate suspicious manuscripts and comb through their own back catalogues for historical problems, with varying degrees of assistance from volunteer sleuths, including those that regularly post on Pubpeer, the global online journal club that is renowned for finding problems in published articles. Despite the commonly held hope that the scholarly record is self-correcting and research fraud therefore has minimal impact, many research integrity officers will say, at least privately, that fraud and malpractice are far too common and legitimate a threat to the record. Some publishers have expressed frustration that other stakeholders, such as funders and institutions, haven’t been more active in this area and that, aside from a few high profile cases, much research fraud goes unpunished. One research integrity officer told me that even when presenting university research offices with cast-iron proof of fraud, they’re often either met with silence, or told that the university doesn’t feel it’s appropriate for them to act.

Tellingly, integrity concerns were heavily reflected in the range of solutions and ideas that were presented on the Innovation Day. That’s not to say that no other issues were being tackled, but research integrity was definitely the hot topic. So in this post, I’m going to break down how the startups and innovators that presented on the day are trying to tackle this particular problem. Broadly, I think they fall into four categories:

• Tools to identify bad content, or sometimes assure good content
• Tools to identify bad actors
• AI-assisted workflow tools to make life easier/quicker for editors and reviewers
• Platforms that integrate workflows and tools

The tools space is evolving in interesting ways. A couple of years ago, when STM hosted the first Integrity Day, tools aimed at identifying problems with individual manuscripts were getting a lot of attention. Startups like FigCheck, ImaCheck, Proofig, and ImageTwin were, and still are,  detecting replicated and altered images; the sort of fraud you can use photoshop to commit (or more likely ImageJ, or GIMP). The STM Integrity Hub had, and still has, a duplicate submission detector, that was based on submission data pooled from participant journals, and the paper mill detector, which, amongst other things, identifies so-called ‘tortured phrases’. These are the odd-sounding sentences that occur when paper mills replace certain words with synonyms to work around plagiarism detection software. For example, ‘artificial intelligence’ might become ‘counterfeit consciousness’. Even then, however, it was obvious that weeding out bad papers one at a time wasn’t going to be enough, particularly in light of rapid advances in generative AI.

Early on, the STM Integrity Hub integrated with the paper mill alarm by Clear Skies, which was one of the first breakthrough tools to take a network-focused approach to identifying bad actors, based on patterns of authors, affiliations, research topic, experimental design, and other attributes. The paper mill alarm won this year’s ALPSP Innovation Award and was a runner-up in the Karger Vesalius award. However, it’s no longer the only game in town. A notable competitor is Signals, which uses a combination of metadata, and network analysis as well as a variety of other indicators, like self-citation rates, previously flagged authors, and community input from Pubpeer. In fact, looking at Signal’s website, it’s starting to look a bit like a platform for integrity tools in of itself, a characteristic it has in common with STM’s own Integrity Hub. Other featured tools included Argos from Scintility, which checks authors and references for retractions, and SciScore, which looks at the transparency and reproducibility of a manuscript based on its content.

The rise in importance and popularity of some of these tools leads to the question of how they’ll integrate into researcher and editorial workflows without creating too much extra burden. Some innovators are taking an interesting approach to this by integrating integrity checks with AI-powered workflow tools designed to make life easier. Reviewer Zero, for example tries to empower reviewers by checking manuscripts for features like missing citations, image manipulation, and statistical inconsistencies. Eliza from World Brain Scholar is a highly-structured platform that takes reviewers through the process, assisting them with AI and retrieval augmented generation (RAG). It even goes so far as to help editors write decision letters to speed the process. Reviewer AI, Veracity, and Charlesworth’s AI peer review assistant, which won the ISMTE People’s Choice Award in 2024, are all taking the approach of supporting reviewers and editors directly.

Ultimately, publishers are going to need to integrate tools into their existing workflows, which in practical terms means integrating into submission and editorial systems. In the meantime, it’s fair to say that all technology providers in the publishing space are thinking about how to support publishers with research integrity needs. According to STM, seven editorial platforms have already integrated with the Integrity Hub, including Highwire, which presented their integration on the day. Others, like River Valley Technology, are independently integrating integrity tools.  Silverchair was also present at the Innovation Day and presented its AI Playground, which is a way to experiment with a variety of AI models, applications, and use cases, many of which are undoubtedly research integrity applications.

After reading this, you probably feel as overwhelmed as I did at the myriad of approaches, startups, and new products working on this challenge. You wouldn’t be the only one. Several publisher research integrity specialists remarked to me that the choice of tools and metrics has gone from a dearth a couple of years ago, to overload. Some even told me that, after implementing platforms that offer large ranges of indicators, they’ve turned quite a few of them off because they’re unsure what they mean or how to interpret them. This move from scarcity to abundance is at least partly due to events like the Research Integrity Day as pressure caused by other factors, like the consequences of mass retractions and high profile cases uncovered by sleuths in places like Pubpeer. In many ways, the challenge of abundance is a good problem to have and indicative of the stage of development of this new function that publishers find themselves with. Over time, it’ll be very interesting to see how these tools and platforms mature and integrate into publishing workflows.

It’ll also be interesting to see whether institutions and funders begin to take more of an active role in policing integrity. Some senior editorial leaders are privately expressing annoyance that publishers find themselves under such heavy criticism at times, while the funders that sponsor research, the institutions that employ the researchers, and even the researchers themselves seem to escape accountability too often. Clearly, it’ll do no good to blame one another, so my hope is that publishers, societies, funders, and institutions can build on the valuable and important work that’s already been done in this space and combine their efforts to ensure the ongoing integrity of the scholarly record.