(This post is based on a presentation given at the 6th annual World Conference on Research Integrity, in Hong Kong, June 2019.)
My objective with this small research project was to get an idea of whether (and, if so, to what extent) articles published in predatory journals are being cited in the legitimate scientific literature.
To that end, I identified seven journals that had revealed their predatory nature when they were exposed by one of four different “sting” operations, each of which had clearly demonstrated that the journal in question will (despite its public claims of peer-reviewed rigor) either publish nonsense in return for payment of article-processing charges, or take on as an editor someone with no qualifications.
I then searched for citations to articles published in these journals in three large aggregators of scientific papers:
- The Web of Science, a massive index of scholarly journals, books, and proceedings that claims to index over 90 million documents
- The ScienceDirect database of journals and books published by Elsevier, which claims to include over 15 million publications
- PLOS ONE, an open-access megajournal that has published roughly 200,000 articles in its history
(Here it’s important to note that Web of Science indexes both the Elsevier journal list and PLOS ONE, which means that findings in either of the latter two databases will represent a subset of the findings in Web of Science. Searching them separately serves the purpose of creating additional context for the Web of Science results, but obviously doesn’t supplement them.)
In the interest of avoiding exposing both myself and my institution to possible litigation, I’m going to avoid naming these journals publicly. Instead, I’ll assign them the letters A through G. I will disclose, however, that six of these seven journals publish in the medical and biosciences — which is, itself, a matter of particular concern.
Journals A, B, C, and D demonstrated their predatory nature by falling for the “Star Wars” sting. In this sting operation, an investigator wrote a putatively scientific paper that actually consisted, in the investigator’s words, of “an absurd mess of factual errors, plagiarism, and movie quotes.” The paper purported to discuss the structure, function, and clinical relevance of “midi-chlorians… the fictional entities which live inside cells and give Jedi their powers in Star Wars.”
Despite the paper’s obviously fictional and even absurd nature, it was accepted for publication in these four journals, thus demonstrating that despite their claims, they do not actually exercise any meaningful editorial oversight or peer review, but in fact will publish anything submitted as long as the author is willing to pay an article processing charge — and will then falsely represent that article to the scholarly world as legitimate, peer-reviewed science.
Another journal betrayed its predatory nature by falling for the “Chocolate Makes You Lose Weight” sting. This sting was perpetrated by science journalist John Bohannon, who put together an actual clinical study of the impact on weight loss of eating one chocolate bar per day. By purposely using a fundamentally flawed research design and subsequently p-hacking the resulting data set, Bohannon and his colleagues were able to make it seem as if their study had demonstrated eating chocolate will make you lose weight.
The article was accepted and published (without a single word changed from its submitted draft, according to Bohannon) in Journal E.
The “Seinfeld” sting followed roughly the same parameters as the previous two, except that in this case the nonsense paper that was submitted for publication purported to discuss the fictional disorder “uromycitisis,” which was invented as part of the story line of a popular American television sitcom. This paper was accepted and published by Journal F.
The final predatory journal under examination for this project was Journal G, which agreed to take on as an editor a fictional individual named Anna O. Szust. According to the fake CV provided by the perpetrators of this sting operation, the fictional “Dr. Szust” had never published a scholarly article and had no experience as either a reviewer or an editor. Not only did this journal accept “Dr. Szust” as a member of its editorial board; amazingly, she is still listed as a member of that board today, even after her nonexistence has been widely publicized. (The word szust, by the way, is Polish for “fraud.”)
Searches were conducted in August 2018 and then repeated in October 2018 as a control. In all cases, the October results were either the same or slightly higher.
Summary of Findings
When I searched Web of Science, Science Direct, and PLOS ONE for citations to articles published in these seven journals, I found that two of the seven (Journal B and Journal D) had never been cited by articles in those aggregations.
Of the remaining 5, it is important to note that Journal E hasn’t always been a predatory journal. It was established in 2008 and published by a highly reputable open access publisher until the end of 2014, at which point it was sold to another publisher, which was the journal’s home at the time the “Star Wars” sting was carried out. While this journal continues to exist and its website shows volumes through 2018, it hasn’t actually published any articles since 2016. These facts will become important in the next section.
The Good News
The findings indicated in Figure 1 are mostly self-explanatory; however, some explanation with regard to Journal E is called for.
Of the three aggregations, the one that contained the fewest citations to predatory journals was PLOS ONE (in which there were no citations to predatory journals except for Journal E—which was cited in 17 PLOS ONE articles, though none of these cited articles was published in Journal E after its sale in 2014).
Elsevier ScienceDirect contained 61 citations to Journal E in total, 31 of them occurring since the sale. However, of those 31 articles in Elsevier journals, 26 cited pre-sale articles in Journal E; in two cases only abstracts were available online, making it impossible to determine the publication dates of the articles cited; in three cases, citations were to post-sale articles. In other words, only 5 articles in Elsevier journals, at most, were found to have cited articles from Journal E that were published after its sale.
The most concerning result was that Web of Science contained 40 citations to post-sale articles published in Journal E.
However, in interpreting this data, it’s again very important to bear in mind that Web of Science claims to index “over 90 million records,” while Elsevier’s Science Direct includes “over 15 million publications”; in both cases the indexed documents include book chapters as well as journal articles. PLOS ONE has published just under 200,000 articles, making its archive a radically smaller data set.
In any case, this table represents the good news: the predatory journals under examination have rarely been cited in legitimate publications indexed by these large compendia of scholarly and scientific literature.
However, there’s also bad news.
The Bad News
Another context in which this data should be considered is that of the predatory journals’ output itself: for example, one of them has had fully 36% of its published articles cited in the mainstream scholarly literature; another has had 25% of its articles cited. Journal E has had only 6% of its post-sale articles cited in the legitimate literature — however, as Figure 2 shows, given this journal’s prodigious output during the two years under examination, that small percentage represents the largest number of articles cited.
And it’s important to bear in mind that this study examined only seven of the most egregious predatory journals in a population of over 12,000 such publications in the marketplace.
Another particularly disturbing aspect of this data is the fact that these three predatory journals all publish in the field of medicine. The fact that these journals’ articles are being regularly cited by articles in the legitimate medical literature should give all of us serious pause.
In conclusion: the data from this study of seven of the most egregiously predatory journals demonstrates the ability of such journals to contaminate the scientific and scholarly discourse. This is clearly a problem.
Other important questions remain, though. These include:
- How big is the problem? What percentage of the articles published by the 12,000 or so predatory journals currently operating represents research that is so deeply flawed that it would have been rejected by competent editors and reviewers? Of those fundamentally flawed articles, how many are being cited in legitimate publications?
- How severe is the real-world impact of the problem? To what degree is bad science, masquerading as good science, undermining the quality of scientific discourse, policy formation, and medical care?
These are urgent questions that call for further research.
The full dataset on which this report is based can be found here.