I wrote a post for The Scholarly Kitchen late last year about the business model for open data. The section that generated the most controversy was this:
…investing in data sharing seems to make the journal less, and not more, attractive to authors. For example, the shift in submissions from PLOS ONE to Scientific Reports coincides with the former’s adoption of a more stringent data policy. As long as authors see strong data policies as just another obstacle to publishing their work, journals will struggle to commit editorial resources to enforcing those policies.
The PLOS ONE story is just a single data point, and many other factors must have been involved – for example, the Impact Factor of their chief rival, Scientific Reports, jumped in mid-2014 from 2.9 to 5.0.
So, setting the PLOS ONE example aside, is there any better data out there on how the adoption of a data sharing policy affects journal submissions? Do authors find these policies so off-putting they take their articles elsewhere, or do they not seem to care?
To get a decent answer to this question, we need to set some constraints on the data we collect. First, data sharing policies vary widely in their wording and intent (see e.g., Springer Nature’s list), and we would expect that milder policies would have less of an effect on submissions. Ideally, we’d focus on a range of journals all adopting the same policy, so that differences in response cannot stem from the wording of the policy.
Luckily, twelve journals across ecology and evolutionary biology adopted the Joint Data Archiving Policy (JDAP) between 2011 and 2014. The policy starts:
[Journal] requires, as a condition for publication, that data supporting the results in the paper should be archived in an appropriate public archive
The policy wording does admittedly vary slightly among the journals, but they do all mandate data sharing as a condition of publication.
The expected effects of a data policy would be i) an immediate fall (or rise) in submissions when the policy was brought in, and ii) a steady decline (or rise) in submissions in the following four or more years. I therefore contacted the Chief and Managing Editors at these journals to ask for their submission data for the four years before and four years after the JDAP was brought in. Specifically, I requested counts of only new submissions (i.e., not resubmissions or revisions) for standard research articles, which should eliminate fluctuations arising from changing editorial policies around acceptance rates and article types.
In no particular order, we* have data from the Journal of Ecology, the Journal of Applied Ecology, Functional Ecology, the Journal of Animal Ecology, Methods in Ecology & Evolution, Molecular Ecology, the Journal of Evolutionary Biology, the Journal of Heredity, Evolution, Biotropica, and the American Naturalist. Molecular Biology and Evolution include the relevant data in their annual editorials.
Standardizing the year each journal adopted the JDAP as year 0 and plotting submissions per year looks like this (to maintain confidentiality the individual journals are not identified):
The first thing to notice is that submissions generally increased: in only 3 of the 12 journals (journals G, I, and L) are there fewer submissions in the year the JDAP was adopted (year 0) than in year -1. In addition, 8 of the 12 journals have more submissions in year 1 than in year -1. There is therefore no suggestion that adopting a strict data policy has an immediate negative effect on submissions. Moreover, the journals that are growing well before year 0 generally continue to grow afterward (journals B, C, D, E, F, and K). Only H and J appear to be growing before the JDAP came in and shrinking thereafter.
We can also do a more formal analysis using a linear mixed effects model for submissions against time relative to the introduction of JDAP, allowing for a curvilinear relationship by including a time2 term. We find that the time2 term is highly significant, and shows that growth in submissions for the ‘average journal’ in this dataset slows 2-3 years after the JDAP comes in. The graph below shows the predicted trend in submissions for this ‘average journal’ and the corresponding 95% confidence intervals.
Did the adoption of the JDAP cause the slowing of submissions growth? This is impossible to test without submissions data from similar journals that did not bring in a data policy. However, we can identify another potential cause: the rapid growth of ecology and evolution articles published in first PLOS ONE (to 2014) and then Scientific Reports (2014 onwards).
Together, these journals put out 18,541 evolutionary biology and ecology articles between the start of 2011 and the end of 2015. Although many of these were probably reviewed and rejected by at least one of the 12 JDAP journals (and are thus included in the submissions data above), a significant fraction were likely sent directly to PLOS ONE or Scientific Reports instead.
More broadly, journal submissions are affected by a wide range of factors that are not considered here – Impact Factors being the main one – and we just don’t have the data at hand for a comprehensive analysis. However, we can be confident that adopting a strict data sharing policy had no consistently negative effect, and that journals that were growing continued to grow once the policy came in. The longstanding fear that journal data policies hurt submissions should now be laid to rest.
*I’d like to thank Arianne Albert, Chuck Fox, Mohamed Noor, Wolf Blanckenhorn, Trish Morse, Dan Bolnick, Anjanette Baker, and Emilio Bruna for help with putting this post together.