Editor’s Note: Today’s post is by Cenyu Shen and Joanna Ball. Cenyu is the Deputy Head of Editorial (Quality) for the Directory of Open Access Journals. Joanna is the Managing Director of the Directory of Open Access Journals.
Throughout their history, scholarly journals have verified the reliability and quality of research, and the peer review process they use is an important building block of trust. In this blog post we reflect on the current threats to trust in scholarly journal publishing, and the implications for organizations like Directory of Open Access Journals (DOAJ) that seek to uphold that trust.
A crisis of trust in scholarly journals
The integrity of the scholarly publishing system faces growing challenges. A combination of factors, including research misconduct, questionable publishing practices, reviewer fatigue, the current culture of rewards and incentives, and misuse of AI technology has significantly undermined the credibility of scholarly journals and cast doubt on the reliability of published research.
Surge in retractions
Instances of research misconduct and other questionable practices are on the rise. In 2023, the number of retracted articles reached over 10,000, exceeding all previous annual records. The surge in retractions is particularly linked to papers published in special issues of journals, often overseen and edited by guest editors. This arrangement can open the door for opportunistic paper mills and other bad actors involved in the sale of authorship and fake papers, manipulating editors and reviewers to facilitate the publication of fraudulent papers. Retracting papers of dubious quality is a time-consuming and resource-intensive process for journals, but is an essential element of healthy scientific process.
Waning editorial oversight
The continued growth of research publications over the years raises a crucial question: can peer review, as it currently operates, adequately manage the growing volume of articles while effectively screening out low-quality submissions? Journal editors constantly struggle to find the qualified peer reviewers needed to maintain the quality of the review process. The increased volume of article outputs places additional pressure on an already stretched system. Whileartificial intelligence (AI) offers the potential to expand the reviewer pool, refine the automated matching of manuscripts with suitable reviewers, there’s little empirical evidence measuring its efficacy. Technology alone cannot fully address the intrinsic motivation problem of researchers’ reluctance to engage in peer review. As the demand for reviewers exceeds supply, there’s a risk of creating incentives to compromise on the rigor of the peer review process.
Reward system pressures
The current “publish or perish” research culture provides an environment that can encourage questionable research practices. Researchers may submit their work to predatory journals as a shortcut to build their publication records, and secure career progression and funding opportunities. This problem is especially worrying in medicine, for example, where many poor quality COVID-19 related papers may contain wrong or fraudulent data for treatment, which has the potential to threaten public health.
Misuse of AI technology
The advance of AI-based technology has introduced yet more challenges to this landscape. Data sleuths have uncovered numerous instances of AI-generated papers infiltrating journals, making it even harder to distinguish trusted from untrusted information. Given the rapid evolution of AI technology, developing a robust tool to effectively detect and prevent such fraudulent practices is an endless struggle.
Trust in journal publication relies on the integrity of editors, reviewers, authors, and publishers. Any deviation from the expected ethical conduct and responsible practices by any of these players presents a significant threat to the foundation of trust and has a profound impact on public confidence in research. As fraud and misconduct has become more common, it sends a warning, signaling that trust in research is already in a state of crisis. Immediate actions are required to restore and strengthen this eroded trust.
Safeguarding trust: DOAJ’s evolving role
The history of DOAJ reflects this increasing focus on integrity and trust within scholarly publishing. Founded in 2003 with only 330 titles, its original purpose was simply to list the growing number of open access journals, and our small team focused primarily on gathering and verifying information. The index grew quickly and by 2013, the total number of journals in the index had hit 10,000. As predatory practices began to grow it became clear that DOAJ’s criteria were inadequate to filter out deceptive journals or those with questionable publishing practices. After a wide community consultation, we developed a new set of evaluation criteria. Every journal in the index was asked to reapply under the new criteria, and as a result, around 3,300 journals were removed in 2016 for failing to reapply.
All journals that apply for indexing in DOAJ need to demonstrate that they meet our current criteria, which include requirements for transparency, adequate peer-review processes, and clear and consistent copyright and licensing terms. Applications are reviewed by a member of the DOAJ editorial team or one of our specially trained community of volunteers, with the final decision on inclusion taken by a DOAJ Managing Editor. Only 24% of applications are accepted at initial application, but many journals use the advice given by our team to improve their procedures before reapplying. If a journal disagrees with our decision, they can submit an appeal. Our Appeals Committee thoroughly examines each case, taking into consideration any additional evidence or compelling justifications provided by the journal.
Due to our rigorous selection criteria and evaluation processes, DOAJ is often cited as an indicator of trust across the scholarly community, used by governments, research funders and institutions such as cOAlition S members, the “Nordic List” of approved titles, and the South African Department of Higher Education and Training’s list. In addition, DOAJ is recommended by many librarians and researchers as a reliable guide to legitimate, reputable open access journals and high-quality scholarly content, and integrated into library catalogues worldwide.
Fostering trust: What we do
At DOAJ, we work actively with the community to improve standards in scholarly publishing and to support editors to improve the quality of their journals. Together with COPE, OASPA and WAME, DOAJ is an author of the Principles of transparency and best practice in scholarly communication. We recently launched the Open Access Journals Toolkit with OASPA to support open access journal editors. Our network of ambassadors, launched in 2016, works worldwide to promote standards in journal publishing and to advise journal editors in good practice. We also partner with Crossref and others to support PLACE (Publishers Learning and Community Exchange), a forum which helps publishers to understand and improve their publishing processes and standards.
The definition of trust in scholarly publishing is not static. DOAJ’s challenge is to uphold trust within a constantly shifting landscape of publication standards and questionable publishing practices. We approach this by refining our criteria, improving our methods, and developing new initiatives.
Our team of sleuths
One important step we have taken to safeguard and maintain the quality standards of journals included in DOAJ is to develop a dedicated quality team. This group works to investigate suspected instances of questionable practices, which have been flagged during the application evaluation process, or by members of our community. The team examines editorial board composition and competence, and checks peer review practices, published articles and a number of other factors. As predatory practices continue to evolve, our investigations are becoming increasingly complex, and we sometimes consult external subject matter experts for their advice. In 2023 alone, we carried out a total of 409 investigations into journals and publishers, many of which resulted in exclusions from DOAJ of at least one year. These decisions are also open to appeal.
Through our investigations, we also see a number of journals that fall within a “grey area” and are low quality, but not predatory. They might demonstrate poor language usage and substandard website design, or lower editorial or production standards. We distinguish these journals from genuinely predatory or deceptive journals.
Refining criteria: endogeny and special issues
We also review and refine DOAJ’s criteria to ensure they adapt to trends in publishing. Two recent examples of changes are the introduction of criteria to cover endogeny and special issues.
In 2021, we introduced new criteria on endogeny as we’d observed a connection between high rates of endogeny and questionable or unethical practices, such as compromised peer review, self-promotion, and potential conflicts of interest. We therefore brought in a threshold for an acceptable percentage of articles authored by editors, reviewers or members of an editorial board. We know that this is challenging for many new journals; it is not intended to penalize credible journals but provide editors with a clear path towards DOAJ acceptance. We work actively with publishers and editors to discuss their journal’s eligibility and to find solutions.
In recent years, we’ve also seen a significant rise in the number of journals relying heavily on special issues as a tool to increase article volumes and the income generated by an APC-based journal. Concerns about the quality of special issues have grown within the community. As a result of these concerns and evidence found from our own reviews, we raised this issue with our trusted partners: COPE, OASPA and WAME. COPE issued a discussion document on best practice for guest edited collections in 2023 and, using this as our starting point, we added new criteria for special issues, which addressed issues of editorial oversight and excluded journals relying solely on special issues. The feedback we had from some parts of our community illustrates some of the challenges we face in implementing criteria that apply across all disciplines and communities. We were contacted by several humanities and social sciences journals and organizations with long-established publishing models based on special issues concerned that the new criteria excluded them from DOAJ. After further consultation, we released an updated version of the criteria without this restriction.
The stakes are high
Both these examples illustrate the unintended consequences for journals and publishers that may arise from changes to our criteria. Over time, our role within the open access ecosystem has evolved to become that of a gatekeeper and an arbiter of standards, and the decision as to whether or not to index a journal can have serious consequences for a journal’s reputation, impact, and even survival.
We take this responsibility very seriously, and are constantly balancing the need to ensure the quality of the journals in our index with our desire to be as inclusive as possible. Our team carefully monitors developments in scholarly publishing and works closely with the community to understand trends and concerns. We take decisions on criteria through a careful, collaborative process, consulting with trusted partners, and working with journal editors and publishers to discuss what is appropriate and reasonable. To increase transparency and to ensure that we are as inclusive as possible in our consultation, we are in the process of establishing an Editorial Policy Advisory Group, which will include global representation of our different stakeholder groups and disciplines.
The future of trust?
Trust goes hand-in-hand with openness and transparency, and it’s vital that the community trust not only the journals in our index, but also our organisation and the work that we do. We aim to be as open as possible, and like many other organisations have made a commitment to the Principles of Open Scholarly Infrastructure. Our criteria, list of journals added and removed, and finances are also openly available.
DOAJ is only one part of a wider network of organisations working to uphold trust in scholarly publishing, and we are encouraged by the move to use markers of trust at all levels across the research lifecycle. Upholding trust is an ongoing process and a responsibility shared by all stakeholders; it needs a coordinated approach where everyone plays their part, from institutional training and support for ethical publishing practices in institutions, to increased transparency from publishers, and to infrastructures like DOAJ which provide metadata and standards.
