AHRQ logoTwo years after the White House issued its memo on Expanding Public Access to the Results of Federally Funded Research” we now have the second of the more than twenty policies to be announced (the first, from the Department of Energy [DOE] came out last August). The Agency for Healthcare Research and Quality (AHRQ) has posted their policy for public access to research papers and to research data, and their approach differs significantly from that of the DOE. Some first impressions below…

For access to research papers based on their funding, the AHRQ will follow the path already blazed by the NIH. Though the NIH has not officially announced a policy, representatives have stated that their PubMed Central (PMC) repository fulfills the White House’s requirements for research papers and will be their method of complying. Given that the NIH is required to use and maintain PMC by separate law, this is not surprising.

Many of the medical and health-related funding agencies are expected to take advantage of PMC for their own public access policies. The AHRQ will require funded authors to deposit their accepted article manuscripts to PMC, to be made publicly accessible after an embargo period of no more than 12 months after publication. It’s interesting to hear the AHRQ describe PMC, a public service of the National Library of Medicine, as a “private-public partnership”. It would be interesting to hear who they think the “private” partners are.

The upside of this approach is that PMC is a known quantity, one that has proven popular and effective in providing access to research papers. PMC is well-established, so many of the basic details of how it will function do not need to be worked out. The downside for the agencies is the cost. PMC is not inexpensive, and agencies will have to pay the NIH for services rendered. Since the federal government is not providing any additional funds for public access, this means that taxpayer money that could have been devoted to funding new research will instead be diverted elsewhere.

Compliance is another big question mark here. We know that the NIH is heavily reliant upon publishers depositing articles on behalf of authors (this report suggests that only 20% of PMC deposits come from authors). Publishers increasingly see PMC as a direct competitor in the marketplace. While I’ve only heard of one major medical publisher threatening to discontinue NIH author deposits, I’ve heard from many publishers that they will not automatically extend deposits for any new funding agency policies. This means authors will likely need to deposit their articles themselves, resulting in an increased burden on the researcher and their institution, and as has consistently been the case where manual deposit is required, reduced levels of compliance.

Like the DOE policy, the AHRQ makes no mention of reuse of articles, particularly for text- and data-mining purposes. The policy also offers a mechanism for shortening the embargo period, but no mention is made of any means of lengthening it, which seems incongruous with the White House policy.

Access to data is vastly more complicated than access to research papers, and accordingly, the AHRQ data policy is more complex. A great deal of the policy outlines how “data” is defined, and like the DOE and NSF policies, researchers will be required to submit a data management plan (DMP). The DMP will then be evaluated by the AHRQ, allowing some flexibility as requirements continue to evolve. The AHRQ expects data to be released no later than the date of “acceptance for publication of the main findings from the final dataset.”

Confidentiality is addressed, and the AHRQ requires “de-identification” to occur, or data to be withheld if it is not possible. The AHRQ will contract with an as of yet unnamed commercial repository, “to accept and manage data submitted by extramural, intramural, and contract researchers.” This is yet another diversion of funds away from new research and it will be interesting to see how well reliance on the commercial sector is accepted by those who favor public repositories.

As far as enforcement, the AHRQ staff, “will periodically manually check applications, proposals, or reports for compliance with the Public Access Policy.” Again this is effort and funding diverted away from research. Manual tracking seems anachronistic as we’re developing better and better automated methodologies for monitoring compliance, and this is an area where I’d like to see the AHRQ policy rapidly evolve. It’s also a bit disappointing to hear that the AHRQ is planning to develop a system of unique identifiers for datasets, which sounds like a reinvention of the wheel given existing standards like DOIs, ORCID and FundRef.

The start date for the AHRQ policy is unclear. At one point in the announcement they note, “Implementation will be prospective and will not apply to any publication or digital data set arising from an AHRQ-sponsored grant, cooperative agreement, contract, or intramural research project funded prior to publication of the final AHRQ Public Access Policy.” At another, they make the policy retrospective, and note that any research funded as of December 2014 will need any papers to be deposited in PMC, and that, “This policy will be effective for research funded in February 2015.” It is also stated that data need not be deposited until October 2015 when a commercial partner has been established. Clarification of these conflicting dates would be helpful.

It’s also interesting that the AHRQ will automatically defer to the policy of any other funding agency listed as involved in a particular project. This seems a smart move, given that there is likely substantial overlap between the NIH and AHRQ funding of projects, and will allow the AHRQ to avoid having to pay when another agency could foot the bill. It will be interesting to see how widespread these deferral policies are, and what happens when two agencies try to defer to one another.

*Post has been updated to correct an initial error regarding the start date of the policy.

David Crotty

David Crotty

David Crotty is the Editorial Director, Journals Policy for Oxford University Press. He serves on the Board of Directors for the STM Association, the Society for Scholarly Publishing and CHOR, Inc. David received his PhD in Genetics from Columbia University and did developmental neuroscience research at Caltech before moving from the bench to publishing.

View All Posts by David Crotty

Discussion

19 Thoughts on "U.S. Agency for Healthcare Research and Quality (AHRQ) Announces Public Access Policy"

Regarding the last paragraph, the publication policy applies to research funded this month, not beginning in October. October is when data deposit begins.

AHRQ says this: “This policy will be effective for research funded in February 2015. AHRQ will solicit contract proposals and fund a commercial repository by October 2015. Beginning October 2015, AHRQ will require all researchers to submit data in digital format to the commercial repository.”

Thanks David. There are several conflicting dates listed in the policy, and I’ve updated the post to note this.

Yes, there are some significant confusions here. The plan seems to say that this is not in fact the final policy, which will be published sometime in the future, but I find no hint what might still be open. I first read this document as a plan to develop a policy, but much of it reads like a policy, so who knows? They also say they have yet to develop contract terms to implement the policy, but there is no mention of a rule-making. It’s a bit of a mess.

And we have this from yesterday to contemplate, as well:

http://archinte.jamanetwork.com/article.aspx?articleID=2109855

http://retractionwatch.com/2015/02/09/fda-repeatedly-hidden-evidence-scientific-fraud-says-author-new-study/

Apparently, the FDA has been allowing studies to be published despite knowing that major methodological or ethical issues exist, up to and including data fabrication.

And the problems go on from there, including hiding a lot of what goes on at the FDA, even from its own advisers, as an editorial published to accompany the main study noted:

“[The author] could not estimate the actual frequency of serious problems owing to the large number of records that were missing or unavailable. And he could not determine how often the violations found by the FDA could reasonably be expected to be acknowledged in an article or how often corrections or retractions were indicated. If the FDA documents had been more accessible and not heavily redacted, however, it is likely that [the author] would have linked more published clinical trials to apparent research misconduct.”

And we had this Sunday night:

Once again, we see that we have much bigger issues than public access to published studies — things like the funding of research, the honesty and integrity of the scientific process, the availability and transparency of oversight functions around research, and how we achieve scientific validity. If any of these break down, public access becomes problematic on a more fundamental level — that is, the public is getting bad information.

Perhaps the AHRQ in particular and the federal government in general should be paying more attention to its core issues and less to peripheral issues.

Kent, are you saying that the design of Public Access is a peripheral issue? If so then I disagree. Interestingly ARHQ devotes a large fraction of its budget to communication of health information, so I imagine they share your concerns about quality, but that does not lessen the importance of access in my view.

I think devoting time to the access issue may be justifiable if wisely spent, but spending money on it with a solution like CHORUS available (and the DOE precedent) is clearly unnecessary. Also, you can spend less time on it by adopting CHORUS, so there is a clear double benefit to it.

I do not think public access is a central issue currently, as we are facing major central threats to research funding, research integrity (intrusive Congress, etc.), and difficulties with reproducibility that is eroding public and funder trust. If you think public access rises to the level of those issues, then we have to agree to disagree.

I certainly agree that we disagree. I regard your supposed central threats as minimal at best. Research funding has held its own through horrendous economic times and all Federal funding, including for research, is controlled by the Congress because we are a representative democracy. As for reproducibility, I consider that to be mostly a bogus scare. But public access is a real threat to scientific publishing and if an agency is doing it badly that makes the issue bigger, not smaller.

Here’s an interesting question: I’ve been told that for any journal that submits NIH funded articles automatically on behalf of authors, that the NIHMS manual system for authors to submit manuscripts does not work. Where does this leave AHRQ authors? If they publish a paper in a journal that automatically submits NIH-funded articles (but not AHRQ articles), then there is no way for authors to upload their articles to PubMed Central.

Is this correct? Does anyone know whether PMC has been updated/altered to fix this?

NASA is also using PubMed Central, according to their policy, which I just noticed yesterday [thanks to a tweet from the @scholarlykitchen account]: http://science.nasa.gov/media/medialibrary/2014/12/05/NASA_Plan_for_increasing_access_to_results_of_federally_funded_research.pdf

NASA say it has been posted online since some time in January. Regardless of when it was actually posted, that now makes 3 agency policies announced after OSTP memo – of which 2 are using PMC.

NASA also commissioned an analysis of their options (CHORUS, PAGES, PMC), and PMC came out on top.

None of these were unexpected, but I cant publicly comment as far as any other agency’s as of yet unannounced plans.

In the NASA plan they repeatedly state that the costs of using PMC will come out of their research funding budget. Also worth noting is this statement, “attractive features of the distributed solutions (CHORUS, Google, and SHARE) warrant further attention in Phase 2 as they mature and are demonstrated.”

Indeed, a PMC-CHORUS hybrid is a long term possibility and this is basically what DOE PAGES is functionally, in that DOE physically ingests what CHORUS does not provide. On the budget side a lot depends on the agency’s scientific communication budget. Unlike DOE, NASA has a big one and USDA has the NAL (including an eleven story building in Beltsville MD). So their Public Access program money is probably not affecting their research funding. On a percentage basis AHRQ has by far the biggest communication budget, roughly a third of their total budget. Communication is a big part of their mission.

NASA states the following:
“Since support for the initiative must therefore be subtracted from agencies’ program budgets,…”

Program budgets yes, but both NASA and USDA have large scientific communication budgets within their program budgets. NASA is pretty spectacular in wowing the public, while USDA has extension agents in every county, plus the NAL of course. I doubt that cost saving is an issue for either here.

One striking thing about all these announced plans is that there is no mention of regulatory rule making. Clearly these public access requirements are new regulations, because they require people outside the agencies to do something, namely submit their articles. Calling these new rules “plans” is very misleading. I think the agencies may be out of compliance with the Federal regulations that govern the making of new rules by Federal agencies (some of which I helped to write). I earlier summarized the rule making rules as they apply to the Public Access program here: http://scholarlykitchen.sspnet.org/2013/10/21/us-federal-open-access-rulemakings-to-come/.

In particular, the rule making requirement to minimize the burden of new rules argues strongly in favor of using CHORUS.

Comments are closed.