Last week, I wrote a long post about a number of things going on at the US National Library of Medicine (NLM), primarily regarding their PubMed and PubMed Central (PMC) initiatives. There are now some updates to share, some further information about how eLife has been treated, and more evidence that PMC can stumble into competition with technology providers.
But first, a recap, with some pertinent updates included.
The three main thrusts of what’s going on at NCBI, PubMed, and PMC are:
- Cronyism — Favoring journals run by friends over journals run by anyone else. The Wellcome Trust, for instance, has had a position on the PMC National Advisory Committee (NAC) since 2005, first filled by Robert Kiley and more recently by the organization’s solicitor. Wellcome is a friend of PMC, and eLife received special treatment (more on exactly how much below). Based on evidence in comments and social media citations, the mirror-image of this cronyism is widely experienced — other journals, including open access (OA) journals, have been following the rules, waiting for indexing, waiting for inclusion in PMC and PubMed, while eLife breezed in, likely because some of the main people involved on both sides know and like each other, and perhaps somewhat because everyone involved works for a powerful funder (Wellcome, HHMI, NIH, or Max Planck). So journals from established publishers have to wait, but those published by cronies of PMC get right in.
- Competition — Favoring certain business models over others, and also competing with technology and its position of authority over the PubMed index. One comment on last week’s post pointed out that the American Physiological Society was treated differently when it decided to publish an OA journal. Suddenly, PMC was interested in facilitating their indexing and inclusion, despite a sometimes contentious past between principals in both organizations. This little bit of evidence underscored the assertion many have made behind the scenes — PMC has a strong OA bias, and is working to tilt the entire biomedical publishing business to OA despite its role as a government agency in a library setting. This bias isn’t as strong as the cronyism around eLife, but it is apparently real.
- Confusion — Exploiting the general confusion between PubMed and MEDLINE to give themselves and their friends advantages in the market. The other aspect that became clearer based on comments and other communications is that the confusion between PubMed and MEDLINE is real, and is being exploited by PMC to great effect. I recall clearly when PubMed was established that it was basically a “port” of MEDLINE into a browser-friendly environment. This is what people still think it is. But it’s not just that. Instead, it’s a separate database of journal titles and content that is assembled by different means, with different standards, and for different reasons. MEDLINE is in it, but more is in it, as well — specifically, titles not indexed in MEDLINE. By tying PMC deposit to PubMed indexing and exploiting this confusion, PMC has given itself and its chosen ones a strong competitive advantage.
The cronyism, competition, and confusion become all the more obvious in the minutes of the last PMC National Advisory Committee (NAC). The meeting was held June 19, 2012, in Bethesda.
The mission of PMC as stated here (it has shifted somewhat since its founding) is:
PubMed Central was established to support NIH’s mission of disseminating the results of biomedical research widely to the public and to the scientific community. PubMed Central employs electronic publishing technology to archive, index and distribute peer-reviewed journal literature in the life sciences. The PubMed Central National Advisory Committee shall advise the Director, NIH, the Director, NLM, and the Director, NCBI, on the content and operation of the PubMed Central repository. Specifically, the Committee is charged to establish criteria to certify groups submitting materials to the system, monitoring its operation, and ensuring that PubMed Central evolves and remains responsive to the needs of researchers, publishers, librarians and the general public.
There’s a lot that can be said about this statement — especially the hard transition of concepts between the first and second sentences, as if one logically follows the other — but notice the wording after “Specifically,” which is the only part of the statement that has been retained throughout the PMC NAC’s existence. It states the NAC should spend time talking about establishing criteria and monitoring. Given these specific charges, you might imagine that the PMC NAC would be concerned with ensuring that its criteria were clearly articulated and uniformly applied. A level playing field is all that’s needed, which a solid, clear, and uniform process should foster. Given the primary objective stated in the first sentence of the missions, you might also find it reassuring if the majority of the discussions centered around information about how effective PMC’s activities have been in improving patient health outcomes, reaching educational goals, and finding favor with researchers and practitioners. There is some of this early on in PMC’s lineage — notably, updates on ClinicalTrials.gov and electronic health records (EHR) integrations (and one interesting section in 2011 about a study showing how ineffective PMC has been with health care practitioners) — but that’s not what the majority of time has been spent on at these meetings. Over the last few years, politics has become an increasing focus of PMC NAC meetings.
Unfortunately, the agenda and minutes of the recent meetings look more like a meeting of OA Central, rather than PubMed Central.
A Special Guest, and Special Advice
Much of the June 2012 meeting was spent hearing from an unlisted guest — Mark Patterson from eLife. At this point, eLife had not even accepted its first paper. It’s worth remembering that Wellcome Trust has had members on the PMC NAC since its early days. It’s also worth noting that having looked through all the minutes from PMC NAC meetings since their inception, nothing like this has ever occurred — a single new publication and an unproven publisher given the spotlight to describe what it’s doing. The videoconference represents a good portion of the minutes, and must have lasted quite a while.
At one point, Patterson said:
The costs of eLife will be totally underwritten by the three funders, at least initially, so that the journal can become established
Yet, just a few months later, a US government agency would become the primary publisher of eLife content, making government subsidy part of their funding scheme, intentionally or not. This bears further emphasis, I believe, because the three funders behind eLife have billions of dollars at their disposal. In a political season such as we have now in the US — with concepts permeating the atmosphere like the “top 1%” and “millionaires and billionaires playing by different rules” thanks to government action or inaction — this has a weird resonance. It might appear that PMC is helping the big funders preferentially, and showing deference to pocketbook rather than proven ability.
Within this conversation, there’s an interesting passage:
Dr. Patterson said that eLife does not plan to charge for an initial period, perhaps three years, and that the longer term business model will be developed over the next year or so. One possibility is to introduce a journal with a high acceptance rate. Another possibility is charging submission fees. He noted that publication fees would not be sufficient to cover costs. Dr. Lipman commented that if eLife was going to introduce a high-acceptance-rate journal, it might be best to do it in a timely way rather than waiting two or three years.
So, not only did the PMC NAC invite an overview of a specific initiative for the first time in its minuted existence — and, specifically, a splashy, well-funded OA initiative — but it also seems that David Lipman was giving business advice. It’s the type that perhaps a meddlesome board member might give, but Lipman is not on the eLife board. Combined with the fact that this was an unprecedented videoconference announcing a single initiative to the PMC NAC and that PMC abetted eLife’s initial publication process a few months later, it’s another sign of both broad and specific cronyism and of boundaries that have been severely eroded.
Boundaries and rules need to exist if we’re going to have a level playing field at the NLM.
Not Recorded, Not Tracked
The lack of concern with process is also striking in other communications I had last week, specifically with Joyce Backus, Acting Associate Director for Library Operations, NLM. I sent her an email the week prior with these specific questions about eLife’s inclusion in PMC and its decision to become the primary publisher of eLife content:
- When did eLife submit their application?
- When did the Acquisition and Selection Section consider their application?
- When did the idea of allowing them to post immediately on PMC arise? Who broached the idea? Who was involved in those discussions?
- How typical is it for the published criteria for inclusion to be waived? When was the last time this occurred? What made you waive them in the case of eLife?
- Are there minutes of the meetings of the NLM Acquisition and Selection Section? Could I have a copy of the minutes for the meeting(s) in which eLife was discussed?
She responded without any specific answers, only to say that these decisions are not recorded because they are done at a staff level; that the reputations of eLife principals factored into the decision; and that the minimum number of articles stated and restated in requirements on their site are more of a “guideline” that they routinely ignore or work around if other factors dominate their thinking.
I restated my questions, looking for specifics. The questions that remained unanswered were:
- When did eLife submit their application? This seems to be an easy one to answer, even if it was handled by staff and not the committee.
- When did the idea of allowing eLife to post directly to PMC arise? Who broached the idea? I’ve heard through the grapevine one version of this story, but would like confirmation from PMC.
- Who was involved in these discussions about posting eLife content directly to PMC? This seems important to know, for transparency and accountability.
Questions 4 and 5 seem to be answered, and in a surprising way at least to me. Essentially, the criteria we read and abide by are much looser in practice than they seem, and are almost whimsically applied. This gets back to the role of the PMC NAC. Why are they not evaluating these criteria and their implementation? Question 5 is answered because there is no real process, just a bunch of staff operating within what appears to be, at best, a loose framework.
The first question matters because Lipman said that eLife submitted their application following “the regular procedure.” If so, there should be a date of submission. It’s a direct question with a simple answer.
Backus responded to my second request for answers in an interesting manner:
My previous email is responsive to your questions in the way I am able to respond. As I noted, selection decisions are a routine part of our daily work and are not recorded committee decisions or tracked processes.
That last sentence only underscores a lack of disciplined process — according to Backus, the process is not even tracked.
For anyone who has filled out an application, sent files, abided by stated NLM submission criteria, and scrupulously followed the process, it’s more than a little anticlimactic to learn that the process is not even tracked.
And, still, nobody at NLM can say how eLife came to be posted on PMC — who was involved in the conversations, who made the decision, when the decision was made.
More Meddling in the Publishing Technology Marketplace
Reading the most recent minutes of the PMC NAC — with a lengthy section from the head of SPARC updating the group on OA legislation, and another long update from Chris Bird from the Wellcome Trust about OA initiatives in Europe — it seems PubMed Central is more concerned with driving the OA agenda through than in managing its own processes to ensure a level playing field for publishers, librarians, scientists, and the public.
In this framework, its role as a technology support for those it favors also goes back a few years, suggesting that its position in the market itself is problematic.
In 2008, when the Journal of Clinical Investigation wanted to move from HighWire Press to independent publishing, the manager of JCI was then a member of the PMC NAC. The motivations for the move were both financial and technological, I learned through email communications. According to minutes from June 17, 2008, PMC helped to smooth the road by providing free XML for the transition, advice on data design, and free hosting of major elements for at least half a year:
PMC helped JCI with the project by supplying its entire JCI archive and by providing advice on data design. The website has been functional since January 15, 2008. There are still some gaps, Mr. Hawley said. PMC is still serving as the host for PDFs and images
This bridge assistance as a technology provider helped JCI leave HighWire without forcing JCI to incur some of the commercial costs for the transition. Typically, moving from one platform to another or to a self-hosted environment requires a publisher to fetch XML from their compositor, validate the XML in the new system, get the PDFs, secure the images, and run systems in parallel at least for a time. It’s a normal cost of doing business. In the case of JCI, PMC provided the XML and advised on its use, potentially stealing business from one part of the publishing technology world, while also hosting PDFs and images for JCI for more than six months, allowing them to leave HighWire earlier than they were ready to leave on their own, potentially robbing HighWire of months of business, as well. While PMC allows its participants to extract their content at any time, it rarely “helps.” In addition, Hawley reported back in on progress to the PMC NAC, making it appear that PMC had a management interest in the transition (Hawley emailed me that JCI did not act at the behest of PMC, but the minutes document interest in the JCI transition at the NAC level).
This effectively made PMC a player in shaping the technology market for JCI — and, again, taking a special interest in those its leadership has close ties with.
The role PMC plays relative to scanning, XML hosting, image and PDF hosting, and online publishing overall needs to be re-examined. It is too competitive for a government agency, and as all scholarly publishing becomes online publishing, this competitiveness hits at too many levels — hosting, archiving, image processing, multimedia, search, usage, indexing, and so forth. The JCI story is just an example of how a basically competitive stance can get tangled up in governance issues and appear somewhat questionable. It would be better if PMC didn’t have any hand at all in solving online publishing for its participants.
Questions linger about PMC’s management and role, questions both large and small. There are questions specific to PMC being the primary publisher of eLife content. There are questions about the process — its rigor, its fairness, its disconnects — around PubMed and PMC indexing and inclusion. There are questions around whether PMC has drifted off its moorings and into OA waters, creating an environment of favoritism and political action that’s inappropriate given its remit. And there are questions about PMC using government infrastructure to play a hand in the commercial publishing technology market.
I hope more answers are to come. Until some changes occur, we will continue to play on a field tilted in certain directions by the NLM, NCBI, and PMC, a field capable of tripping up some of us and creating shortcuts for others, one that is managed opaquely and rather poorly by those who represent themselves as the officials in charge, and one that poses as disinterested but which pushes some interests, mostly toward supporting the OA business model and perceived “friends” of PMC.
21 Thoughts on "PubMed Central or OA Central — More Strange Behaviors at PMC and NLM Paint a Portrait of Biases and Poor Process"
Keep up the good work, and keep digging.
If the depiction of eLife from the minutes is accurate, it’s disappointing to hear how scaled-back the vision for the journal has become. As I recall, it was originally pitched as an OA competitor for Science, Nature and Cell, using the formidable economic power of two top funding agencies as a means of overcoming the financial difficulties that have kept other attempts at selective, top tier OA journals in the red. Now if they’re instead considering just creating another PLoS ONE clone or going the submission-fee route, the journal seems more mundane, and less the radical experiment many were hoping to see.
The initial period where everything will be free has now been defined (originally suggested at 5 years, now apparently 3), and it’s pretty much the same strategy employed by nearly every new journal on the market, whether OA or subscription-based, make things free for a few years to establish usage and reputation, then move to the journal’s actual business model. Has there been any mention of the current state of some of eLife’s other floated economic innovations, including paying editors and peer reviewers for their work? Are those still on the table?
Perhaps a lesson eLife is teaching us is that the same economic pressures apply, whether you’re PLoS, a commercial publisher, a society journal, or a very wealthy funding agency, and that those pressures drive journals to the same few approaches for viability.
I agree it’s far less exciting if it basically goes with the PLoS model rather than the funders subsidizing publication. Indirectly they end up funding publication anyway, at least for their own funded research by paying the APCs. It would have been nice if they stuck with the original model. Obviously with a few challenges but also a lot of benefits and one that makes sense at least to me.
PMC from it’s inception has been focused on creating a freely accessible archive of research articles in biomedicine. That is it’s mission. All the articles in the archive can be accessed in an HTML format that is generated from the XML. If the publisher chooses to supply a PDF, they will serve that too. That is part of its role whether you like it or not. If eLife or some other journal chooses to use that as it’s published version, so what? That option is available to any journal archived in PMC. It is irrelevant and changing nothing that NLM is doing.
As far as I know PMC has no set limit on the number of journals/articles it will house. It is not a situation where “my journal gets in at the expense of your journal”. PMC requires the journals they accept to meet a quality criteria and demonstrate the ability to provide XLM with an accepted DTD. They are flexible on specifics of these criteria for everyone. They were for me in getting a journal accepted. The staff told me it was flexible and while the guideline stated they would like 50 articles for the XML test we were only able to provide about 30. That was not a problem. It was just a guideline. After a couple rounds and a few tweaks to our XML they were satisfied. We had no “cronies” to help us and as best as I can tell we got no special treatment. As far as the quality evaluation, there is nothing on their webpage describing the process that indicates a minimum number of articles. Perhaps they informally circulated a guideline of 15 articles as stated in your earlier post but I suspect that is flexible. The documentation does not state any set number of articles.
PMC is not Medline and Medline is not PMC. Selection for Medline is peer-reviewed and rigorous because it is manually indexed. It’s not some big conspiracy, every journal in PMC is indexed PubMed but it is done digitally. The journals in Medline are assigned MESH headings manually. If you search on MESH heading you only get a search of Medline journals. If you do free text searches you get results from PMC as well. I am sure the NLM did it because it just makes sense. Manual indexing is expensive, computerized indexing is cheap. Indexing PMC makes the literature much more accessible. That’s that idea.
Of course PMC is OA centric. The whole idea is to make the contents of the archive freely available to anyone. It always has been. It may not fit well with the business model your journal uses, but supporting that business model is not NLM’s role.
There are at least two so-whats. First PMC is not supposed to be in the original journal hosting business. This is not archiving. Second it means their published criteria are smoke and mirrors. If they host the journal then the journal cannot have the required track record before they host it. I am amazed that you do not see the problem. This is a classic case of mission creep by PMC.
Since its inception, PMC has drifted into an OA stance. “Freely available” is not the same as open access, and peer-reviewed articles have become their focus over initially stated materials such as “screened but not peer-reviewed reports” and the like. It is now regularly receiving reports on political activities regarding OA from SPARC, and is working with splashy OA journals as outlined.
If PMC requires journals to meet quality criteria, why are those criteria being applied inconsistently and in a way that suggests favoritism? As for the documentation, refer to the original post, which shows they state 15 articles and reiterated this via email to another publisher on the same day Lipman was telling me about their flexibility.
PMC is not MEDLINE, but you’re confusing the issue with new confusion. I never said there was confusion between PMC and MEDLINE. The confusion is between PubMed and MEDLINE. A lot of people — authors, readers, publishers — think MEDLINE and PubMed are the same. You’re saying NLM created this confusion because “it just made sense”? I see that as illogical. Manual indexing is expensive, but that’s what people presume is going on.
As for PMC being OA-centric and that being OK, there is a difference between creating a level playing field for free content based on NIH funding requirements and current laws, and creating a tilted playing field based on cronyism, loose standards, lax processes, and clumsy operations that betray an underlying bias in the agency. What if PMC were subscription-publisher biased by having Elsevier, Wiley, and Lippincott on its Board and special guests from those publishers updating them on their pet initiatives, which then would receive tacit PMC support through technology and indexing services? Would that be OK?
NLM, PubMed, MEDLINE, and PMC should be unbiased, reliable, fair, uniform, and transparent. The NLM should be an agency that helps create a level playing field. What’s wrong with that?
The E-BioMed proposal included both a pre-print archive and a peer-reviewed section where the NIH would run the peer review process. The whole thing would have been open access and while pre-Creative Commons, the proposed licensing looks similar to CC-BY.
That got shot down by all the pressure from commercial publishers and societies that felt threatened by the proposal. The “drift” in creating PMC was actually in the other direction, toward just archiving existing content.
PMC works with all journals and if they meet the basic criteria they get into the PMC archive. Do you know of a good quality journal that has asked been refused?
I take what is presented on their web site as the criteria for selection rather of some letter to a publisher. Can you provide what was said in the letter? Did they say 15 articles was a specific criteria or a general guideline? It would seem what they publish on their web site would be the official criteria.
The goal of indexing is to make it easy to find the published literature. Medline was never designed to provide some rating system for journals. Medline is selective because it is expensive to manually index journal articles and the belief hand indexing does a better job of allowing someone to find exactly what they want. If you just want to search Medline, you can just use MESH terms. If you want to search Medline plus PubMed you can do that via free text searches. If you dig around in the instructions for PubMed, I believe that is documented.
A level playing field for who? The mission of the NLM is to make the biomedical literature more accessible. PMC is designed to do that. NLM isn’t obligated to make it fit with your particular business model.
Sorry, I intended to say “If you want to search Medline plus PMC you can do that via free text searches.” rather than PubMed.
The PMC requirement is: “The journal must have a reasonable number of published articles in order for NLM to make a decision about its scientific quality.” Note that eLife could not have had any published articles prior to being accepted.
In emails, PMC personnel routinely state that the minimum is 15 articles.
A journal needs to be included in the NLM Collection to meet PMC’s scientific quality standard. This review can take place once [the title] has published 15 articles.
We confirmed this just before the post went live, and received the response I published in the first post:
One of your new journals (it doesn’t matter if it is OA or not) will need to publish at least 15 articles before it can be evaluated by the Selections Group.
If they require articles to be published before evaluation, and state multiple times in direct communications that the minimum is 15, what are you to believe? And if eLife gets a free pass on the pre-publication requirement AND the number of articles? Please.
MEDLINE indexing is widely regarded as a standard of legitimacy and quality, and is now routinely confused with PubMed indexing. Creating an OA back door into PubMed indexing is wrong, because it fools people. It’s that simple.
For the vast majority of researchers, there is no functional difference between PubMed and MEDLINE, and there remains a tremendous amount of confusion as to where PMC fits into things. I can’t tell you how often I hear researchers ask whether a journal deposits NIH-funded papers into “PubMed”. And I know several publishers who take advantage of this confusion by depositing articles into PMC to give authors and readers the illusion that the journal has been accepted for indexing in MEDLINE. Some do this as a shortcut, to attract authors faster without having to wait for the formal MEDLINE review process, others do it as a workaround, knowing that their journals are not going to be accepted into MEDLINE due to low quality, they deposit in PMC, knowing that for nearly all readers and authors, the difference is moot. The only thing that matters to the authors/readers is whether the paper turns up in the results of a PubMed search. That’s it. That means the paper is “in PubMed”, end of story.
The question then, is whether there remains any reason for the MEDLINE selection process. If, for all intents and purposes, acceptance into PMC provides the equivalent of being indexed by MEDLINE, then why spend the time, effort and money necessary for the more demanding review process?
The question is whether there is a need for manual indexing. Probably not since the vast majority of searches are just free text. At the same time should NLM not index PMC just because some people use it in a way it was never intended to by used?
Manual indexing is of vital importance for professional information scientists to perform precise and comprehensive searches. Medline indexing is some of the best in the world. Getting rid of it would be a huge injustice.
Some of the language you use is pretty harsh, but I do think that the committee may be overstepping its role as a government function. While I’m not against inviting guests for information purposes, there are other instances where the committee is collaborating with their guest. In the June 2011 meeting, they are clearly giving advice to researcher, John Willinsky, on how to set up his experiment:
The Committee discussed [John Willinsky’s] proposed study design and suggested various options to consider, such as narrowing the study population to a specialty such as internists or pediatricians in order to get a statistically valid result, and studying a population of health care providers that no longer has access to the literature but previously did.
You can decide what is to be implied by the phrase, “in order to get a statistically valid result.”
I’ve been to meetings where the fear of collusion (or at least the perception of collusion) drives several members to speak out with general warnings about what cannot be discussed. Considering that the NLM is required to make its minutes public and encourages transparency, I would think they would be more careful about what is said at these meetings.
You’re absolutely right, the language is a little tougher than you’d see in some venues, but it’s quite tame compared to what I’m hearing from people who have been affected by this over the years. There is no love for PMC and its practices outside of a cozy group. George Carlin would be proud of how well I’ve implemented the FCC seven filter.
Your point about the cautions most boards or committees exercise is exactly right. Most responsible boards or committees know their boundaries, worry about exceeding them, and are run by a chair or designate who monitors those boundaries. The lack of perceived boundaries or accountability in the PMC meetings is striking.
“There is no love for PMC and its practices outside of a cozy group”
Kent – you need to get out more. PMC has provided millions of US taxpayers with access to the results of biomedical research funded with their tax dollars, articles that would otherwise be prohibitively expensive or unavailable. I’d wager that there’s a lot more love out there for PMC and the National Library of Medicine than there is for commercial publishers.
If the general public has any love for PMC (and there’s little evidence of this), it would likely only be because PMC is presenting for free the fruits of commercial and non-profit publishers’ efforts in refining the literature.
But that’s not what’s wrong with your comment. The context of the phrase you pulled was “within the publishing community.” Please don’t take the quote out of context.
Even people who you would think might be supportive of PMC are expressing disappointment at least, with some verging on outright anger, over how PMC has managed itself and how its staff present themselves within the publishing community. PMC is not doing its jobs in a neutral, fair, or even-handed manner. That’s the problem. And they have fewer friends than they think, within the publishing community, even among OA and other publishers, who find their behaviors bizarre and mercurial. I don’t think that’s how a government agency funded by our tax dollars should work.
Kent from a perusal of your comments and analysis I see a reflection of the current ideological bent in the US, namely: 1) It is good to have friends in high places; and 2) rich or poor it is good to have money.
Something really smells rotten here. I would suggest sharing this information with the staff of the appropriate committees of Congress and let them look into the matter in connection with their oversight responsibilities.
Won’t it be grand when the government controls all scientific publishing! It worked so well in the Soviet Union.