Editor’s Note: Today’s post is by Andrew Grey, Alison Avenell, and Mark Bolland. Andrew is a clinical endocrinologist and Associate Professor of Medicine at the University of Auckland. Alison is a medically trained clinical biochemist and a Professor at the University of Aberdeen, Scotland. Mark is a clinical endocrinologist and Associate Professor of Medicine at the University of Auckland.
Consider these facts:
- A meta-analysis that reportedly screened >20,000 titles is completed and submitted to a prestigious international medical journal in under 4 months. Concerns about its reliabilityare immediately raised. The journal verifies the concerns but takes no public action for more than 1 year.
- A journal is notified that the author of research it published has fabricated clinical trials. Subsequently, detailed concerns about the publication integrity of 10 papers are raised with the journal. The editor declines to become involved, and defers to a publisher staff member, who repeatedly fails to answer requests for updates about the concerns. Almost 6 years later none of the concerns about the 10 affected publications has been resolved or readers alerted to the concerns.
- A publisher says it will only assess the integrity of papers authored by the same fabricator when concerns are raised on a publication-by-publication basis (personal communication, Elsevier publishing staff).
- A journal publishes a letter detailing concerns about internally inconsistent results for the primary outcome from a very influential publication. The authors decline to resolve the concerns, and the journal concludes that nothing more can be done and fails to link the paper to these concerns.
- A national research regulatory body investigates widespread concerns about more than 150 publications: it determines that a university committee, featuring several co-authors of the publications in question, should make the final decision(s) about the outcomes for the affected papers (personal communication, Iran National Committee for Ethics in Biomedical Research). Three years later, to our knowledge, there is no publicly available report or details about any of the investigations.
- An academic institution investigates concerns about the integrity of publications by a staff member: it fails to detect or act upon the absence of ethics committee approval for several studies.
- A World Health Organization-approved clinical trials registry accepts unquestioningly a slew of changes to 119 retrospectively registered clinical trial documents which were made shortly after concerns about the published trial reports are raised, and are designed to ‘resolve’ widespread discrepancies between trial registration information and trial publications. Registry staff do not reply to notification of this. When journals and publishers are notified, they do not act.
- A leading academic in osteoporosis admits to falsifying data in multiple clinical studies, after trying for a long time to cover it up and deflect the blame elsewhere. Only 4 papers are retracted. She is barred from clinical practice but then reinstated 2 years later. Meanwhile, at least 112 of her publications remain in the literature without any indication as to whether they are reliable. A co-author of one thought it had already been retracted because it was unreliable. The relevant Universities did not investigate these papers, and her medical college insisted it was not their responsibility to consider information not part of its disciplinary proceedings.
In each case, readers of the affected publications remain unaware of the problems, and the damaging ripples of the flawed papers distort the literature and clinical practice.
No one questions the critical importance of a reliable biomedical literature. Universities teach research integrity, publishers espouse it fulsomely, government agencies debate and endorse it. There are definitions, recommendations, and guidelines. Yet the cases described above are but a few of the many examples of the slow, opaque, inconsistent, frustrating, and unsatisfactory outcomes of tumbling into the rabbit hole of publication integrity. Watching paint dry is ultimately more fulfilling: at least the paint will be dry eventually. Why is achieving and maintaining publication integrity so fraught? Could it be that the main protagonists don’t actually care?
Publishers have the final say in publication integrity, since they decide what to publish and what to correct. Publications are their ‘product’. Most organizations selling products have mechanisms to ensure the quality of their wares. Assessment of publication integrity should be part of that process at publishing companies. Yet even large enormously profitable publishing houses seem to invest little in the practicalities of resolving integrity concerns. Instead, they may devolve responsibility to journal editors who often have little experience, interest, time, or resources. Publishers rarely collaborate with each other in resolving concerns. Publishers usually do not undertake a comprehensive review of publications by researchers with track records of misbehavior and multiple retractions. When decisions about integrity concerns are reached, publishers rarely report the nature of the concerns, the processes undertaken to assess them, or the responses to them. So, the most important parties – readers of the publications and members of the public whose health is affected by them – remain ignorant of the concerns. All of this suggests publishers don’t care.
Guidance for addressing concerns about publication integrity suggests, very reasonably, that the publisher seek a response from the authors and, depending on the circumstances, an investigation by the authors’ institutions. However, most integrity concerns about publications can be assessed without input from institutions, whose contribution is in any case largely confined to determining researcher behavior, i.e., whether ‘misconduct’ occurred. In correspondence with us, publishers frequently complain that it is difficult to engage with authors and institutions, which suggests that neither of the latter really cares about resolving problems in publications. Although there are notable individual exceptions, many publishing staff seem willing to allow assessments to drift along in a torpid state, repeating the mantra that they are “following the COPE guidelines” while declining to report the status of their assessments. Presumably they hope that eventually a resolution will present itself. Or perhaps that the problems will just disappear. Maybe they think that the research they so profitably publish is of so little importance that it doesn’t matter? It certainly suggests that they don’t care.
The Committee on Publication Ethics (COPE) and its guidelines are often referenced when publication integrity is at stake. COPE seems a strange organization. After beginning as an ad hoc collection of concerned journal editors, COPE has evolved into a charity (definition: an organization set up to provide help and raise money for those in need) whose funding is by subscriptions from publishers and member journals, so it is essentially industry-funded. Membership is restricted to those with direct links to journals and publishers. COPE is therefore strongly journal/publisher-focused. It sets out guidance without involving other important parties (e.g., readers) and focuses on journal processes, not methodology, for integrity assessment. If a journal/publisher claims it followed COPE guidelines, the organization does not appear to question the outcomes. COPE’s guidelines are sufficiently ambiguous that journals and publishers who claim to follow them reach quite different conclusions when considering very similar concerns about publications from the same researchers. COPE makes no recommendations about timeframes for resolution of concerns, and does not sanction members who do not follow its recommendations, perhaps because it lacks either the authority or the will to do so. In our experience, COPE’s responses to requests from journal readers for advice and assistance are erratic, unhelpful, and anonymous, in that they are conducted entirely via an administrative staff member. While it has undoubtedly made some important contributions to publication integrity, COPE shows little sign of evolving into an inclusive organization, with journal readers’ needs foremost, that facilitates timely and rigorous assessments of publication integrity rather than merely the appearance of doing so.
Faced with concerns about an employee’s research, institutions often appear to care first and foremost for their reputations, not the integrity of what is published under their aegis. Commonly, they conduct secretive, prolonged, and incomplete internal investigations, focused on damage control and determining whether ‘misconduct’ occurred, rather than whether all the publications in question are reliable. The reports of their investigations are seldom made public, nor shared with publishers. The key affected parties, readers of the publications, are almost never made aware of the existence of an investigation or its findings. That suggests academic institutions don’t care about publication integrity.
If the leading actors in this grim drama really cared about the reliability of published research rather than their bottom lines — profit and reputation — they would acknowledge the serious flaws in the current approaches to publication integrity, and commit to improvement. But they haven’t. The current processes are so opaque that it is impossible to be reassured that strategies to meaningfully improve the assessment and resolution of publication integrity are even being considered. Academics who work in the field, freely performing the role of quality control officers for publishers, are still treated with suspicion and disdain. The implicit paternalism (“don’t worry, we’ll handle this, but we won’t bother you or our readers with the details”) of publishers and academic institutions is surprising, considering the issues at stake. Perhaps each has forgotten that flawed publications impact adversely on the health and wellbeing of those who fund, participate in, and conduct the research that publishers rely upon for their existence. Or perhaps they don’t care.
Is it harsh to suggest that key organizations do not care about publication integrity? Some may argue that egregious incentives, such as career advancement from quantity and self-defined ‘impact’ of publications, and the competitive business model of academic publishing fuel compromised publication integrity. But many of these incentives are sustained or driven by institutions and publishers, who therefore can influence them.
Improvements are clearly possible. Publishers can review and publish new submissions within a few weeks so should also address integrity concerns in a timely fashion, rather than awaiting the deliberations of lengthy ‘misconduct’ investigations. Publishers could also require more of authors at manuscript submission: agreement to participate in any investigation and/or provide raw data should concerns be raised about the publication, and agreement that failing to participate will lead to withdrawal of the publication. Publishers could also require and audit raw data, support the establishment of independent panels to assess publication integrity, collaborate with academics with relevant expertise, and invest more appropriately in quality control. Institutions can disengage from perverse incentives to publish, and audit research practices among their staff. Both can commit to transparent, non-conflicted, comprehensive, and efficient assessment and resolution of concerns about publication integrity. Such sensible improvements, however, require that each protagonist actually cares. Based on our experience, it is not clear that they do.
18 Thoughts on "Guest Post — Who Cares About Publication Integrity?"
It would be interesting, in cases where the researcher is not yet tenured, or fully promoted through the scholarly ranks, to see a study into whether their colleagues care. Or, are they recommended by their colleagues for tenured/promoted anyway?
A great piece! Publishers should also recognize that, for some, writing is an art form. Not only “le mot juste” but entire sentences and paragraphs. Having had agreeable referee reports, an editor required me to change the wording of a short paragraph from one of my own papers, that I had cited. It had been picked up by the journal’s anti-plagiarism software and the editor wanted it neutralized in that respect. I appealed to COPE, but the editor remained intransigent. The paper went elsewhere. Subsequently finding the case-history in COPE files defeated me.
I hope COPE will reply. I have never been involved in the organisation but have always looked up to it. The idea that is a secretive organisation is a surprise to me. I have had a different sort of problem with anti-plagiarism software. As a publisher I worked with an excellent editor who would summarize the papers in an issue in a short “preface”. The software revealed that he was using material from this content without full attribution. He was summoned by the dean of his faculty. I was called to assist the editor. I found that the dean perceived the editor as doing something very wrong and was in danger of being dismissed. I argue on contextual grounds but to no avail. An editorial essay is not a peer reviewed paper. The editor had to give up his useful preface to the issues to keep his job. It was a loss to readers
Important source to better understand – consider checking out: Ending Medical Reversal. Improving Outcomes, Saving Lives” by Vinayak K. Prasad, and Adam S. Cifu. https://www.press.jhu.edu/books/title/11308/ending-medical-reversal
And on methodological issues, this is one of the best and most concise summaries I have come across: https://sensiblemed.substack.com/p/study-of-the-week-choices-oh-my-the
It’s difficult enough for editors & reviewers, but imho the velocity and politicization of social media mentions re: COVID health research in particular broke things even more.
As scholarly publishing changes from a 19th C. model of the publisher as editor with ultimate control of the integrety of the journal, to an article based platform model where the publisher really is just the administrator of the content platform, with less and less to do with the actual content.
Authors must have the ultimate responsibility for their content – both in owning the rights and the responsibilites. Editorial boards need to have a much happier retraction (trigger) finger. That will cause issues of its own as cries of ‘cancel culture’, censorship and freedom will abound: but remember, the author has many ways to make material available that isn’t part of the pre-publication peer review system.
Fewer articles, fewer journals, harder thinking in review, an open pre-print culture. Unfortunately that attacks the big publisher’s profits, so is likeley to be an unpopular opinion here.
Unfortunately that attacks the big publisher’s profits, so is likeley to be an unpopular opinion here.
I’m always a bit fascinated by comments like this. Why would one assume a blog written by three librarians, 5 people who work at not-for-profit organizations, 9 independent consultants, 6 people who work for non-profit research societies, the CEO of a non-profit publisher, someone from a research institution, and two people working at vendors for publishing services would be strong proponents of the profits of big publishers? Upon what evidence is that assumption based? One of the complaints we usually get when we survey our readers is that we don’t have enough representation of the viewpoints of commercial publishers. Do you regularly read this blog?
The evidence for that view is that you haven’t published any posts that extol views that are quite as radical and extreme as my own. And no, I won’t write a guest post, please stop offering to publish it. /s
My critique of the scholarly kitchen is around it’s deep investment in the systemically racist western scholarly publishing discourse.
To think that the main concern from your readers is that there is no voice for the billion dollar elite publishing corporations is quite revealing in its own right.
The thinly veiled attack on COPE in this article is an exemplar of the this kitchen’s approach. Any journalist would have made an attempt to contact members of this poorly funded group of well meaning people who have put a huge amount of energy into attempting to address the issues in this article. Instead, a brief review of the website was all that was managed.
This site is the opinion of a group of privileged academics and businesses, and needs to be reviewed through that lens, and called out for the corporate shills the cooks don’t even now they are.
Well that escalated quickly! Thanks for the response.
Two corrections and a request for clarity:
1) You make a mistake we commonly see about this opinion blog, thinking of this as “journalism” rather than people sharing ideas and opinions in their spare time. The closest it comes to “journalism” might be an editorial page in a newspaper, where when someone writes an opinion piece that criticizes something/someone, they do not routinely reach out to the object of criticism for a counter-opinion. Our doors are open, however, to COPE should someone from the organization care to respond, either here in the comments or with their own post.
2) The lack of corporate voices was not “the main” concern of our readers, rather one of many.
I would agree that we are largely writing from positions of privilege — it’s something we’ve acknowledged frequently and has been part of our struggles to bring better representation to the blog. Our authors need to voice strong opinions, something those who are more vulnerable are unable to do without consequences. To help with that we do offer the ability to write anonymous posts (e.g., https://scholarlykitchen.sspnet.org/2018/04/04/excluded-testimonies-people-color-scholarly-publishing/ ).
On the racism charge, I’d stand by our record of being leaders in bringing attention to those issues in our community (see https://scholarlykitchen.sspnet.org/category/diversity-and-inclusion/ ) but I do agree that there is much more that needs to be done.
I am confused a bit by your accusation that we are deeply invested in the discourse of scholarly publishing, yet you then call us out for criticizing that same establishment by offering space to guest authors who see one of its core resources as ineffective. Are we “shills” or are we unfair attackers? The statements seem contradictory.
I am, however, still waiting for evidence of our support for “big publisher profits”.
The reason that these issues don’t get addressed is, in my opinion, a simple one – nobody is willing to accept _ownership_ of the problem. If the problem isn’t “yours”, whoever “you” are (authors, publishers, the academy, COPE, whoever) there is little incentive for “you” to work to solve it. There may be great interest among such players in _having_ it solved, but until some body or group of bodies is willing to take ownership of these issues, they’re not going to get addressed in any sort of ongoing and systematic way.
Defining a problem like this can be pretty simple and straightforward. But _solving_ it is another matter entirely, especially if none of the players, whether singly or in come way collectively, are willing to own the problem. Lack of ownership is likely to continue to lead to only tepid interest among the primary players in contributing, in any sort of concerted way, to such a solution.
Thank you SK for this guest post. Hope some stakeholders comment with evidence that they do care about publication integrity.
Thank you Andrew Grey, Alison Avenell and Mark Bolland for summarizing your evidence that they do not. There are cases of good editorial and institutional practice out there but they are, perhaps, not enough to undo the damage caused by widespread inaction, negligence and inefficiencies.
If you’re interested in this subject and haven’t yet read it, this is an interesting suggestion: https://www.science.org/doi/full/10.1126/science.ade3742
Thanks Andrew Grey, Alison Avenell, Mark Bolland and SK for the excellent post. It is a mess, and even when a journal editor does the right thing and moves forward to remove prima facie fraudulent articles without waiting for a stonewalling institution, it can be a huge job to do a fair and accurate investigation, such as the Jonathan Pruitt affair (https://www.nature.com/articles/d41586-022-02156-2). What I don’t understand is why journals aren’t more liberal with editorial Expressions of Concern. If for example, an author has a paper retracted for falsified data, putting an EoC on other papers by the same author alerting readers would seem an obvious step.
P.S. One lesson from recent research integrity scandals in ecology (Pruitt, Danielle Dixson affairs) is the only real bulwark for publication integrity is for journals to enforce data publication. It’s still quite difficult to convincingly fake genuine data.
We thank the authors of the recent Scholarly Kitchen guest post ‘Who Cares About Publication Integrity?’ for their thoughts and comments around the frustrations that are experienced when attempting to preserve the integrity of the scholarly literature.
COPE is a UK-registered charity (https://publicationethics.org/about/governance) whose objectives are to to “educate and advance knowledge in methods of safeguarding the integrity of the scholarly record for the benefit of the public”. We aim to do this by promoting and hosting collaborative dialogue, creating publicly available resources such as position statements, guidelines, discussion documents and flowcharts, and providing a space for important conversations to be aired. COPE is not a regulatory or statutory body and we do not have the mandate to force our members to take a certain course of action. It is worth noting that COPE has a published sanctions process (https://publicationethics.org/cope-sanctions-process-flowchart) for its members but our approach in such cases is firmly in the direction of education through mediation and discussion.
Our Trustees and Council Members hold regular subcommittee, Council and Trustee Board meetings throughout the year to reflect on and discuss issues of concern within the academic publishing community, and to formulate plans to support and move these on. The agenda for our annual, in-person meeting later this year will also include some of the topics raised in the guest post..
It is important to note that COPE is part of a wider ecosystem that upholds the integrity of the published record. We work with research groups, scholarly societies, publishers, journals, editors, other organisations and individuals – and more recently, universities and research institutions – across all academic disciplines and geographical locations in order to promote best ethical practices in publishing. We believe that by working together we can raise awareness of these issues, promote high standards of ethical behaviour, and attempt to produce guidance and advice that supports changes in behaviour, policies and processes to promote publication integrity. We know that there is much work to do to achieve COPE’s specific vision (https://publicationethics.org/about/cope-strategic-plan) of “creating a future in which ethical practice in scholarship is the cultural norm” and we will continue to listen to and work with the research community in this shared endeavour.
To address some of the more specific COPE-related items in the piece:
1. When an issue of concern is raised to the attention of the Facilitation and Integrity subcommittee (https://publicationethics.org/about/subcommittees#Facilitation), by either members or non-members, our role is to facilitate a discussion (https://publicationethics.org/facilitation-and-integrity-subcommittee) that we hope will lead to a resolution, increasing transparency and a move toward a more ethical environment. The remit of the Facilitation and Integrity subcommittee is reviewed regularly to consider some of the limitations inherent in the process. If the authors of the guest post have specific feedback they would like to share with us then we would be happy to receive that directly (https://publicationethics.org/contact-us).
2. COPE membership (https://publicationethics.org/become-member): COPE started out (25 years ago) to support academic editors; however, it is open to all those working in the field of publication ethics, including individuals. We have also recently launched membership for universities and research institutes (https://publicationethics.org/membership/universities-research-institutes) with the aim of bringing together journals/publishers with universities/research institutes to strengthen the network of support, education and debate in publication ethics.
3. COPE guidelines are usually generated from our discussion documents (https://publicationethics.org/guidance?t=&type%5B%5D=Discussion+documents&sort=score). Discussion documents review complex issues with various opinions about appropriate solutions and actions and we always seek community feedback after publication of discussion documents. We welcome comments on any of our resources at any time and these are considered by the Education subcommittee for further updates to the resources. The vast majority of COPE resources (https://publicationethics.org/guidance) are publicly available.
4. All of COPE’s work is carried out by a small, paid support team and a volunteer Trustee Board and Council. We are grateful to our Trustees and Council Members who work tirelessly in their own time to help shape best practice in ethical publishing, and support the community with advice and guidance.
It’s must to put the case studies on the table of researchers associated with the university and research institutes to sensitizing them.
We thank those who responded to our guest post, and offer the following in reply to posts that were directly relevant to our piece.
George Duimovich suggests that discussions on social media might exacerbate the difficulties faced by editors in dealing with publication integrity. On the other hand, they might be helpful, by drawing attention to concerns – more rigorous research is needed to address those possibilities. It is hardly surprising that some commentators use social media to raise concerns about publication integrity, given the problems we described in doing so via ‘conventional’ mechanisms.
We were surprised to read Anton Angelo’s description of our work as a “thinly veiled attack on COPE”, when only a small amount of the post is dedicated to that organization, and when we wrote that it “has undoubtedly made some important contributions to publication integrity”.
We agree with Mel Desart that there is need for a better and more democratic approach to improving publication integrity, including involvement of journal readers and those whose wellbeing is affected by flawed publications. That, however, requires that those who broker the current system ‘accept ownership of the problem’, acknowledge the system’s inadequacies and commit to real investment in improvements. It is not evident to us that such is the case.
We acknowledge Karen Shashok’s point that there are “cases of good editorial and institutional practice”, which can serve as exemplars for best practice. Our experience, from many more examples than we were able to describe, is that they are uncommon. We have found that cases of good practice are led by committed individuals, rather than a result of robust systems.
Chris Mebane suggests it is a “huge job to do a fair and accurate investigation”. That may be true when researcher behavior (“misconduct”) is being assessed, but considerably less so if the primary focus of the assessment is on publication integrity, which is the much more important issue for readers of the literature. Assessing publication integrity is comparable to peer review, which many biomedical journals request be done within 10-14 days. It can and should be promptly undertaken first, before investigation of how integrity was compromised. We think this a critically important, but seldom practised, aspect of ensuring reliability of published research. We agree that editorial expressions of concern are not applied sufficiently frequently, quickly or consistently.
Natalie Ridgeway’s response, on behalf of the Committee on Publication Integrity (COPE) trustees and Council Members, highlights some of the organization’s shortcomings and overlooks some of our comments. Our experience, repeated several times and implied also in the post from Donald Forsdyke, is that ‘collaborative dialogue’ with COPE does not occur. We might be outliers in this regard – perhaps COPE might survey correspondents with academic expertise in publication integrity to reassure that “collaborative dialogue” is indeed the norm. It might also improve its transparency by making available to correspondents the contact details of the committee member(s) who apparently review concerns that are raised. It is indeed worth noting that COPE has a sanctions process but we wonder what has to happen before it is invoked. We acknowledge that COPE is “part of a wider ecosystem”: several key components of that “ecosystem” were the focus of our piece, in which we provide evidence that it is not performing well in its aim to uphold publication integrity. We were struck by the irony of the invitation to provide specific feedback via the same communication channel that resulted in erratic, slow and unsatisfactory responses to several previous requests and enquiries.
More importantly, the response from COPE does not address the serious issue that the outcomes from journals and publishers that purportedly follow its guidance in addressing concerns about publication integrity are bewilderingly inconsistent. We acknowledge that such guidance documents are well-intentioned, but their application is inconsistent and does not seem to serve the intended purpose. As we point out, this may well be because the guidance is focused on journal processes for communicating with authors and institutions, rather than the more important matter of how publication integrity should be assessed. Although COPE might not regard itself as a ‘regulator’, journals and publishers frequently consider that following the COPE guidelines is synonymous with adequate assessment of integrity concerns. Perhaps these really important issues might be addressed by COPE, especially given its close relationship with publishers and their integrity staff.
Finally, and most importantly, the response from COPE largely focuses on defending the status quo. We hope our piece might contribute to a robust review of that status quo, perhaps in COPE’s ‘annual, in-person meeting’, from which the ‘wider ecosystem’ might benefit. If COPE does care about publication integrity, isn’t it concerned about the frequent examples in which the wider ecosystem fails, and shouldn’t it be motivated to do its utmost to improve matters?