What is PubMed? Is it a search engine? A credentialing system? A filter? Is it a publisher? An enabler of open access (OA) publishing? A technology provider? A competitor? Depending on the situation, you can answer each of the above in the affirmative, despite the contradictions many of the combinations may cause.

Twenty years ago, PubMed was a credentialing system, an online port of the MEDLINE index. This shift of medium quickly made it a search engine, but one built on a manual and highly curated index. If your journal was in MEDLINE/PubMed, it had gone through an exhaustive evaluation, and had earned a badge of legitimacy. You were searching a credentialing system. You were getting filtered results based on MEDLINE inclusion criteria, which were well-accepted.

laptop user asking for help

Starting with E-Biomed and stretching to PubMed Central, the credentialing system pegged in many users’ minds morphed into a publisher (a primary publisher in the case of at least two journals, and, for a time, three), a competitor (here, here, and here, as well as here and here), and a technology provider. Its management also has advanced the cause of OA publishing, in both legitimate and controversial ways. Part of the extension of PubMed’s original purpose has come through brand extension via PubMed Central, which has been used to muddy the waters about what is PubMed and what is not. PubMed inclusion has been used, for example, as an incentive for journals to include themselves in PubMed Central, eroding the idea that PubMed is a neutral arbiter of quality.

Now, a new twist is emerging, and that seems to be that PubMed may be consciously or unwittingly acting as a facilitator of predatory or unscrupulous publishing.

In a paper published in Neuroscience, the authors analyzing the neurology and neuroscience journals included in PubMed found that:

  • Twenty-five predatory neurology journals were indexed in PubMed, accounting for 24.7% of all predatory neurology journals.
  • Fourteen predatory neuroscience journals were indexed in PubMed, accounting for 16.1% of all predatory neuroscience journals.
  • Only one of the 188 predatory neuroscience or neurology journals appeared in the DOAJ index.
  • Only 54.6% of the journals deemed predatory in neuroscience actually contained articles.

The authors note how PubMed Central provides a backdoor into PubMed and the National Library of Medicine (NLM) for these predatory publishers:

Noteworthy, predatory journals are retrievable in the catalog of the National Library of Medicine (NLM), are labeled by a unique NLM identification number and referenced as “Not currently indexed for MEDLINE. Only citations for author manuscripts are included.”

This is the kind of fine print that will escape the attention of most users, and which itself is unclear as to reliability, process, or value. Are we to think that something that appears in PubMed and the NLM but not in MEDLINE has some lesser status? The confusion is multi-layered for users.

This study was published in April, but went largely unnoticed until the authors published on 19 August 2017 a letter based on the same research in the Lancet. In this letter, the authors state their recommendations thusly:

Furthermore, although the National Library of Medicine refers to these journals under the descriptor “Not currently indexed for MEDLINE”, citations for author manuscripts are labelled as “included”. Thus, highly regarded databases like PubMed and PubMed Central should raise the bar for journal acceptance, and join the Directory of Open Access Journals, Scopus, and MEDLINE in imposing stringent criteria for inclusion of journals and publishers.

As one blogger notes about the study and the letter in the Lancet:

The PubMed database managers have irresponsibly allowed it to become a repository of citations to predatory journal articles. Among other things, the next time you see a questionable journal proudly announcing that it is indexed in Pubmed, chances are that the journal is predatory. Contrary to the popular notion that only genuine and distinguished journals which take peer-reviewing seriously and follow all the norms of scientific publishing are indexed in PubMed, many predatory journals too are included in PubMed. The same holds true for PubMed Central too.

PubMed’s brand has long been muddled in ways that pass lower-quality works through the system under cover of prestige. This has real consequences. A recent article in BusinessWeek outlines how the “predatory” publisher OMICS is being used by pharmaceutical companies to quickly get citations for studies major journals won’t accept and run meetings to push their products.

Despite being caught red-handed publishing “fake science” in Canada, OMICS doesn’t seem fazed. In response, it is running today’s standard playbook built on bluster and bombast, including calling accusations against it “fake news” and its founder claiming implausibly, “We’ve never made a mistake.” Currently the subject of a US Federal Trade Commission prosecution, OMICS:

. . . first got into trouble with the U.S. government in 2011. The National Institutes of Health, a division of the U.S. Department of Health and Human Services banned OMICS journals from indexing in PubMed Central, one of the world’s primary databases for medical research, given “serious concerns” about its practices. In 2013, HHS accused OMICS of trademark infringement and using the names of employees at the NIH and PubMed Central “in an erroneous and/or misleading manner.”

While OMICS is ostensibly banned from PubMed Central, it doesn’t take long to come across the OMICS Journal of Radiology in PubMed and PubMed Central. This again shows what a confusion of journals PubMed has become. One URL you get on the first page of a simple Google search for OMICS in PubMed is “https://www.ncbi.nlm.nih.gov/labs/journals/omics-j-radiol/” with the curious “labs” element in it. At the top of the resulting page, it reads, “A PubMed Lab experiment.” Clicking on this tagline takes you to a blog entry from September 7, 2016, which talks about this new experiment apparently aimed at letting users follow certain journals. It seems like an experiment that never took off, yet a Google search finds the page, an OMICS journal with a full accoutrement of PubMed elements is discoverable, and PubMed citations are easily retrieved.

PubMed’s brand has long been muddled in ways that pass lower-quality works through the system under cover of prestige. This has real consequences.

You may also have caught that the reporters from BusinessWeek were also confused by what PubMed Central is, describing it as “one of the world’s primary databases for medical research.” PubMed describes itself as “a free full-text archive of biomedical and life sciences journal literature.” Yet, you can find articles about quantum chromodynamics (QCD) from the European Physics Journal C, Particles and Fields, in PubMed Central. There are hundreds of entries for astrophysics, many of which seem irrelevant to biomedicine and life sciences. Several US funding agencies also use PMC as their repository, including the Department of Homeland Security and the Department of Veteran Affairs. It seems “biomedical and life sciences” are quite broadly defined by PMC.

Maybe PubMed is confusing even to itself?

In addition to the dubious role of certification databases when it comes to questionable journals, the role of scientists in the predatory publishing phenomenon is the subject of a recent article in Nature by David Moher, Larissa Shamseer, Kelly Cobey, and their colleagues. In their review of nearly 2,000 biomedical articles from more than 200 journals thought to be predatory, more than half of the corresponding authors hailed from high- and upper-middle-income countries using World Bank criteria. This flies in the face of the common assumption that predatory publishers are exploiting desperate authors in low-income countries. More worrying is that of the 17% of the sampled articles that reported a funding source, the NIH was the most frequently named funder. The US itself produced more articles in this sample than any other country save India. The authors note:

Our experience with these journals is that they provide both poor vetting and poor access. Their websites and archiving systems are unstable. Although some articles appear in PubMed (often after a delay), the titles are not indexed by Medline and are difficult to find.

What’s the thread through PubMed over the years since it was ported from MEDLINE? The unifying theme I see is a hunger to adapt. At times, this has been a boon — the port of MEDLINE to PubMed was a smart move, and some interface changes have been commendable. At other times, these adaptations have revealed a clear lack of purpose and mission, such as the controversial involvement with eLife, the competition with publisher brands and traffic, and now the loose standards that have allowed unscrupulous publishers to enter PubMed via PMC.

Adaptation is required to remain relevant, but there have to be limits, or the adaptation may cause the entity to simply dissipate into the environment via entropy. PubMed seems to be giving into entropy. It needs to realize this moment calls for something else — clarity, standards, and credentialing that means something. Their opportunity is not to follow, but to lead.

Kent Anderson

Kent Anderson

Kent Anderson is the CEO of RedLink and RedLink Network, a past-President of SSP, and the founder of the Scholarly Kitchen. He has worked as Publisher at AAAS/Science, CEO/Publisher of JBJS, Inc., a publishing executive at the Massachusetts Medical Society, Publishing Director of the New England Journal of Medicine, and Director of Medical Journals at the American Academy of Pediatrics. Opinions on social media or blogs are his own.

View All Posts by Kent Anderson

Discussion

26 Thoughts on "A Confusion of Journals — What Is PubMed Now?"

As we’ve come to expect from your posts, a thoughtful and thorough account of an important topic for the scholarly communication community. Thank you.

It seems to me that once Pubmed decided to compete with commercial databases quality went out the window and quantity entered through the door.

Thank you for writing about this. I became aware of the backdoor back in 2011 when a researcher friend told me about an article one of his post docs co-wrote in a journal not indexed in Medline. He needed that article for NIH purposes so he told me how he just submitted it into PMC and was able to use it because it now had a PMCID. As a librarian I was surprised at the time that this happened and I lamented about the possible findability of the article because it didn’t get indexed. It wasn’t until a year or two later when Beall and his list of possible predatory journals became popular did I start to see the possible problems PMC could have for PubMed. For annual reviews my institutions now uses publication lists derived from Web of Science because they are interested the quality of the article and the journal. While many may disagree with the specific metrics used for impact factor at least it is a metric. PubMed does not have a metric for assessing quality of the article and PMC’s open submission has watered down the journals found in PubMed.

Given the current situation, what would you recommend as the future direction of PubMed? Can it implement new (or reimplement old) standards, or would it need to build a new brand–a subset of curated literature from the heap that PubMed has become?

I think they’ve confused things so much that they need a harder reboot. In that regard, I think their brand framework would support a re-emergence of MEDLINE as a quality mark. They could keep PubMed and PMC as their database brands, and put MEDLINE back atop the heap as their quality measure brand, their certification brand. However, NLM’s brand management has not been very good over the years: https://scholarlykitchen.sspnet.org/2013/02/14/extension-and-conflation-how-the-nlms-confusing-brands-have-us-all-mixed-up/

I am commenting from a librarian perspective.

This is a problem: however, even articles in reputable journals can be faulty. Peer review is not perfect.

Evaluation is critical — knowing how to search for and read articles: Critically reviewing methods, results, conclusion/recommendation, references, funders/conflicts of interest (such as for-profit corporations), etc.

I have encountered undergraduate and graduate students that are fearful of questioning studies in journals because they are written by someone with a PhD or affiliated with a elite institution.

I regularly show Retraction Watch — http://retractionwatch.com — which includes a variety of real examples: plagiarism, faulty research, and more.

Agree completely. However, credentialing is an aspect that can be involved in critical evaluation. When a credentialing service has changed how it behaves, people need to take that into account.

Interesting article. It just prompted me to read the ‘NCBI Website and Data Usage Policies and Disclaimers’, and I found this interesting tidbit: “Liability: For documents and software available from this server, the U.S. Government does not warrant or assume any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed.” One could argue that it was never the intention of PubMed to become a credentialing tool, but rather a pure repository. As John mentioned above, the responsibility of evaluating the literature rests with the reader.

Considering that PubMed ends with a .gov suffix, the only other function it should serve is as a policy tool – it is maintained by the NIH, which funds most of the public biomedical research in the US, after all – and I think much of the confusion stems from PubMed’s ever changing role since it was established. However, speaking as a foreign researcher that peruses PubMed quite extensively, methinks that it is doing a fairly good job as a repository of scientific literature.

I wouldn’t over-interpret standard legal boilerplate into a philosophy or stance.

As I detailed, PubMed began as an online version of a credentialing service, and has morphed into something rather hard to describe over time. In your last paragraph, for example, you describe it in at least three different ways.

The inclusion in PMC of all research articles funded by NIH is mandatory and, I believe, a wonderful advance. Are some such articles trash? Certainly. But they need to be in the repository, and it’s silly to have them there without indexing them in PubMed.
(Are some articles in Science or Nature trash? Also yes.)
A few NIH-supported articles in PMC don’t mean the journal’s content as a whole is indexed. I know if at least one case in which NLM asked a (legit but new) publisher to change the wording on its web page from “indexed in PubMed” to something like “archived in PMC,” so I think they’re trying to keep the distinction clear.
Personally I’ve never had trouble understanding the NLM catalog’s “not currently indexed in MEDLINE; selected author articles …” language. I never had the idea that PubMed or PMC was meant to be, or claimed to be, a gatekeeper or a stamp of quality.

Parsing this comment a little more closely confirms a number of points of confusion — PubMed and PMC crosstalk creating confusion as to whether one or the other serves as a catalog, index (credentialing service), or archive.

The idea that PubMed or PMC was never meant to be, or claimed to be, a gatekeeper or stamp of quality belies the history of both, which includes PubMed emerging from an index of quality with strict inclusion criteria (MEDLINE), operating for years as equivalent and gaining the reputation as equivalent as far as indexing (note that a journal mentioned in this comment wanted to be listed as “indexed in PubMed”), and has resisted losing this credentialing status even as it morphed into a downstream registry of articles in PMC (a technical and brand conflation NCBI used in conjunction with eLife to advance that journal’s fortunes in its early days). Now, PMC itself is not limited to biomedical content but is being used by Homeland Security and other governmental agencies, adding to the documented confusion among editors, authors, and journalists.

If you’re not confused by this cat’s cradle of objectives, brands, functions, and roles, I applaud you.

Good point. With major funders like NASA using PMC as their repository, the “Med” in PubMed Central is increasingly incorrect.

You wrote, “The idea that PubMed or PMC was never meant to be, or claimed to be, a gatekeeper or stamp of quality belies the history of both, which includes PubMed emerging from an index of quality with strict inclusion criteria (MEDLINE).”

You are interpreting inclusion of a journal in MEDLINE as a quality measure, but did NLM ever actually state that that was its intent? I have been using MEDLINE since way before PubMed, and I only ever interpreted it as a computerized index to articles (abstracts). I had always assumed that their reluctance to index anything before it had published N articles or lived for M years was simply a reluctance to commit indexing time and subscription costs to a publication that might prove to be short-lived.

Let’s quote the NLM directly to answer your question:
https://www.nlm.nih.gov/pubs/factsheets/jsel.html

Scientific merit of a journal’s content is the primary consideration in selecting journals for indexing. The validity, importance, originality, and contribution to the coverage of the field of the overall contents of each title are the key factors considered in recommending a title for indexing, whatever the intended purpose and audience.

MEDLINE is described as the NLM’s “premier bibliographic database.” Journals are selected for inclusion in MEDLINE by “the Literature Selection Technical Review Committee (LSTRC), an NIH-chartered advisory committee of external experts analogous to the committees that review NIH grant applications.” Applying for and receiving MEDLINE (or PubMed) inclusion has always been a hallmark and validation of journal legitimacy and quality. This changed when PMC created a backdoor into PubMed.

If you want to read a potboiler of how PMC, NCBI, PubMed, and the NLM got tied into knots trying to find loopholes in their rules and reputation system in order to help a friend, I’d recommend this: https://scholarlykitchen.sspnet.org/2013/02/06/answers-finally-how-pubmed-central-came-to-help-launch-and-initially-publish-elife/.

If you want the whole sordid series, this: https://scholarlykitchen.sspnet.org/tag/freedom-of-information-act/

A very thought provoking article which challenges us all in the health professions where the order is that the ultimate credential for in career recognition of publication is a PubMed number. Is it time for review and revision in term of academic values in health professions publishing

For me, Pubmed and PMC are very clear. They are the biggest repository. Similar to Sci Hub, which is more inclusive. I went myself through the process of sending manuscripts from yet unaccredited Journals. I think it is just a bureaucratic procedure and nothing else. Personally, I think open access is way to go and old monsters like Nature and Science should give way. Especially as many of these “top” journals are not truly peer reviewed. The first stage is editorial review, which is more like market assessment. These are big boys clubs journals with frequent very poor quality science. Yes many journals, like Omics are outrageous, but many others that were initially labeled as bad, are evolving and becoming good now. They will prevail.

I thought that (and recently posted to HIFA) that there was a window of time during which PubMedCentral was more or less taking all comers, until they changed the policy on this in 2014 to effectively close this loophole (as stated in the answer to the FAQ Who may contribute to PubMedhttps://www.ncbi.nlm.nih.gov/pmc/about/faq/#q14) Am I wrong?
I wondered if the issue of articles from poor quality or “predatory journals” appearing to be PubMed records because they are PubMedCentral records was rather a question of publication bias: suppose you are an NIH-funded investigator, and your study shows negative findings. Your preferred journal rejects it (because of publication bias against negative results) but you are mandated to put your manuscript in PMC as a condition of NIH funding, and even “author manuscripts” must be accepted by a journal, so you publish it wherever you can to comply with funding requirements, and then get on with your work.

People like success stories, and the boring reality of scientific research needs champions. Otherwise no-one gets excited about negative results, even though they are of equal value!

Or is this like closing the barn door after the horse is gone? Just asking.

https://www.ncbi.nlm.nih.gov/pmc/about/faq/#q14

In 2014, with the approval of the PMC National Advisory Committee, PMC implemented a scientific and editorial quality review procedure whereby expert consultants from outside NLM conduct an independent review of journals seeking to participate in PMC. This was in response to a significant increase in new publishers and journals applying to participate in PMC, many of which are unknown to NLM in terms of quality and publishing practices.

The independent review follows an assessment by NLM that the journal meets NLM’s criteria for its collection, as outlined in the Collection Development Manual. NLM’s Library Operations Division takes the independent reviewers’ opinions into account in making the final decision on a journal’s suitability for inclusion in PMC.

The problem is that there are very different standards for being listed in PubMed Central (https://www.ncbi.nlm.nih.gov/pmc/pub/addjournal/) and for a journal being indexed by MedLine (https://www.nlm.nih.gov/pubs/factsheets/j_sel_faq.html), yet results presented by PubMed show both and make no differentiation between the two. This is compounded by many years of PubMed only showing MedLine indexed journals (which requires more stringent criteria for acceptance), and many don’t realize that this is no longer the case.

I think the details are always worth considering, but the big picture here is equally important, if not moreso. MEDLINE and PubMed and PubMed Central are now, I think, hopelessly tangled or smudged brands. We don’t know at a glance what each does, how it does it, and what it stands for. Some people want to exploit this, some are trying to correct this with what I think are half-measures. It might be time to reboot the brands overall.

The quality Medline citations are there in PubMed but a disappointment to me is that the status of the different record types in the database is increasingly buried and only visible in the abstract view with a status tag of [Indexed for Medline]. Previously the status tags were visible in the brief citation view. https://www.nlm.nih.gov/pubs/techbull/ma17/ma17_pm_update.html Of course if you search using MeSH terms, you will only be searching citations that have been indexed for Medline. It is possible to exclude unindexed citations too using the various citations status subsets – https://www.ncbi.nlm.nih.gov/books/NBK3827/table/pubmedhelp.T.status_subsets/

The term ‘indexing’ is problematic if it is used to describe both citation metadata AND also the value-added subject indexing of titles that have been selected for full Medline inclusion.

It would be helpful if NLM distinguished in the brief citation format between citations ‘indexed’ because they have met the NLM’s criteria as a Medline journal and thus have MeSH indexing as well as the purely bibliographic author, title, source, abstract type of citation data. Of course recent publisher supplied and in process citations are a mix of citations – some that will end up becoming Indexed for Medline citations and some others that come in from PubMed Central and any other sources. It should be possible though to tag citations coming in from journals chosen for Medline and tag them with a field such as [will become a Medline citation] or the like.

The Neuroscience paper that claimed that PubMed ‘indexed’ some non-Medline papers was using ‘indexing’ to mean only the citation metadata – not that they were indexed for Medline – the NLM Catalog makes this clear – for example – https://www.ncbi.nlm.nih.gov/nlmcatalog/?term=101622222 “Not currently indexed for MEDLINE. Citations are for articles where the manuscript was deposited in PubMed Central (PMC) in compliance with public access policies. For further information, see Author Manuscripts in PMC.”

I agree though that you have to have extensive knowledge of the intricacies of PubMed to be clear on what you are seeing in the retrieval and it should be possible to make some of these distinctions clearer to a lay-searcher.

The perception of authority granted is the end-result of much of this confusion.

I am glad someone (Cheryl Hamill) mentioned using the MeSH thesaurus to search PubMed. I believe NLM has long selectively indexed some high quality journals that include biomedical literature (eg, Nature). Because the citations are submitted by the journal publishers, articles outside the scope of MEDLINE sometimes appear in the results of key word searches, but not in the results of searches using the thesaurus. Key word searching may sometimes still be needed if very current results are needed, as there is a time lag between receipt of the latest citations and the subject indexing of the articles being completed.

I agree that it would be useful for articles not indexed using MeSH terms to be better distinguished. I also think Kent’s insightful and thought-provoking post makes a case for increased training in the use of PubMed.

Comments are closed.