A notable science journal implements a new author service: For $5,000, the publisher will make that article freely available (OA) to the world upon publication. A substantial number of authors purchase this service, either by choice or because they were required to do so by their funding agency, and the program is deemed a success.
Several years later, the publisher is interested in knowing whether those articles made freely available performed any better than articles moving through the traditional publication route–in this case access by subscription. The publisher hires a consulting firm, gathers citation data at a particular point of time and makes a simple comparison between the OA group and the subscription group.
The results show with remarkable clarity that the OA group significantly outperformed the subscription group. Does that mean that OA was the cause of the difference?
Maybe. But before OA is considered to be a cause, the analyst needs to rule out other potential explanations for the effect. Authors elected to pay $5,000 for the OA service, and that choice alone may signal that there is something very different about the OA group compared to the subscription group. Specifically, the OA article group may:
- Represent researchers with better access to research and publication funds than those in the subscription group
- Represent research funded by an organization that mandates–and pays for–OA publication
- Represent researchers located at institutions (or labs) that generate high-profile work, or
- Represent research performed by high-profile researchers
There are many attributes of papers that are hard to quantify, like novelty, relevance, and impact. These are characteristics that may only reside in the minds of readers, but they are associated with things that we can measure, such as funding and location and prior publication history.
Last week, the Nature Publishing Group (NPG) released a report on the citation and download performances of research articles published in Nature Communications–an NPG journal that provides authors with the option of making their article open access (OA) for a fee. The research was commissioned by NPG to the British Research Information Network (RIN) and posted on Figshare. Nature widely promoted the study and even gave the RIN analyst her own blog post.
The results confirm what many uncontrolled observational studies have reported since 2001, when Steve Lawrence published his provocative letter in the journal Nature–that there is an association between freely-accessible research and citations. The cause of the association is not clear, but it has been widely assumed that free access improves access, which leads to an increased chance of discovery and citation. The RIN researchers report:
Overall, articles published OA appear to show a higher number of citations, though the effect is small, and the data provided does not allow us to control for possible confounding effects such as the posting of articles in repositories, the number and location of authors, and the possibility that authors are selecting their ‘best’ papers to publish on OA terms. Similarly, any effect of OA on the timing of citations appears to be small, and we have not been able to control for possible changes such as increased awareness of the journal on the part of both readers and authors. But although the impact on citations is small, the impact of open access publication on HTML views and PDF downloads is large and significant, suggesting increased visibility for the open access papers.
After more than a decade of research on this topic, why is the research community no farther along in settling the access-performance question? Why are publishers still drawn to this question, yet remain unable to provide much evidence or insight clarifying the causal nature of the relationship?
The problem with answering this question is largely methodological. The vast majority of these studies are observational, meaning that the researcher does not attempt to provide an intervention (such as assigning free access to one group of articles) but relies on authors to make their own choices. As a consequence, the researcher is fundamentally unable to distinguish OA effects from self-selection effects. For the RIN/Nature study, OA may provide no beneficial effect with the exception of stratifying authors into those who are able and willing to pay $5,000 and those that are not.
There is a methodological solution to dealing with multiple and confounding sources of causation, and that is the randomized controlled trial. By randomly assigning papers into an intervention (free access) and control (subscription access) group, you can ensure that the groups are largely equal, in all respects, at the beginning of the study, with the exception of the intervention. I did such an experiment on hundreds of papers published in dozens of journals and, while not a perfect study, gets us much closer to isolating and measuring the OA effect.
Even without attempting an intervention, there were ways for the RIN analyst to check whether the OA citation advantage could be explained by other causes. She could have looked at the funding sources associated with the paper, the location of the research team, or the h-index of the first and last author. These data are available in the Web of Science–the source of the citation counts used in this study. The analyst could have asked NPG to indicate which articles were rejected by Nature (or Nature specialist journals) and referred to Nature Communication. As the sample size was small, a simple Google search would have turned up personal, departmental and lab pages where an article was made freely-available, or whether it was deposited in PubMed Central or an institutional repository. Without attempting to gather any additional information, we are left scratching our heads as to whether the citation difference was caused by open access publication or merely associated with open access publication. We are no closer to knowing the answer than in 2001.
Still, those promoting this study use language that suggests that open access is the cause. For example, RIN executive director, Michael Jubb, stated for the Times Higher Education that the results added to “the growing body of literature showing that open access is good for article citations and, especially, online visibility.” NPG’s own press release makes a breathlessly bold claim followed with a hint of hesitation:
It’s clear to see that the effect of open research on citations impacts all levels of research positively. We realise that this doesn’t definitively answer the question of whether open access articles are viewed and cited more than subscription articles, but we think this contribution adds to the debate.
After 13 years of research on this topic, are we really still at the debate stage, or is “debate” just a positive spin on a study that promulgates confusion and ambiguity on the topic?
Apart from the marketing surrounding this study, there are other details in the report that concern me. First, the analyst makes a simple comparison based on just one citation observation (total citations, accrued in April 2014), for all articles, despite the differences in their ages. The analyst acknowledges that a more “fruitful and accurate” analysis would have compared articles at a certain point in their publication lifecycle (e.g. at one and two years), but makes no effort to gather these data [Note: you can extract yearly citation counts from the Web of Science]. Similarly, the analyst was provided with the exact publication date for each article, but ignores these data for simple categorical breakdowns. I’m not even confident that the analyst understood how to code the data, the test she used or how to interpret its results.
It appears that the analyst got OA and subscription-access papers mixed up. Figure 5 (should that be Table 5?) reports median citation differences between OA and subscription-access (Subs) papers. Overall, OA papers received 11 citations compared to just 7 citations for Subs, a difference of about 4 citations in favor of OA. The OA-citation advantage is consistent across all subject types. However, in Figure 6, the analyst reports negative z-scores for each subject, suggesting that the effect benefits Subs over OA. The study also reports an effect size of -.16, suggesting that OA papers performed worse off than subscription access papers.
I’m not sure why the analyst, the project coordinator at RIN or those commissioning the study at NPG didn’t pick up on the disparity between the text and the tables, or why the effect size goes in the wrong direction. You don’t need to be a professional statistician to pick up the apparent contradiction.
Similarly, the main results of the study–that the citation effect is small–are not supported by the data. The effect is rather large, in fact–about 4 citations across all papers in the study or nearly 60% more citations in favor of the OA group.
This study relied not on a sample of articles from Nature Communications, but the entire population of articles–every single research article published from 2010 through 2013. With the entire population of articles, it is not necessary to make a statistical inference from the data. All that was really necessary was some descriptive statistics–like median and interquartile comparisons. Unfortunately, the analyst put more confidence in the statistical test than on the simple comparison. When a statistical test returns some wacky or counter-intuitive results, there is usually something wrong in the data or in the statistical model. This should have set off some red flags with the analyst or those that reviewed this report.
I appreciate that the NPG/RIN agreed that the data should be openly available for debate, but this seems like a feeble excuse for a poorly-analyzed study. Those at NPG who commissioned this study should have demanded a lot more from the RIN and missed another opportunity to tackle the access-citation question. While this report was not treated as a peer reviewed journal article, perhaps it should have been, and met the rigorous standards for design and reporting that the NPG demands for any study that is worthy of its brand.