If you were privy to attend the UK Parliament’s Business, Innovation and Skills Committee hearing on open access, or spent the good part of your morning watching the archived video, you’ll note how economic and scientific studies were used to further one’s position and to undermine the position of another.
You’ll also note that anyone with a PhD in the hearing was addressed as “doctor” whether or not they were acting in the role of scientist, activist, politician or industry representative. Somehow, completing graduate school conferred lifetime authority to any person speaking on any topic.
Beyond the dramaturgy of UK Parliamentary theatre, I became intrigued in how the context of nuanced studies were stripped away to make simple declarative findings, and how complexity in science was used to make negative categorical statements, like “there is absolutely no evidence that…”
For this post, I’m going to focus on a single study, PEER (Publishing and the Ecology of European Research), and how this study, and its findings on article downloads, was retold in the first hearing.
Cameron Neylon, Director of Advocacy for PLoS, used the PEER study to argue that open repositories posed little risk to publishers (video time: 10:27:25). Indeed, archiving actually drove readership back to the publisher.
The PEER study was a large study of commercial publishers that deposited manuscripts of different ages into four institutional repositories. It was a randomized controlled trial that made half of the papers “visible” and the other half “hidden.” Hidden, in this case, meant unavailable for three months from one institutional repository — the reader was provided a link to the same paper located at another repository.
If you’ve read through many of the PEER reports, getting this study running was no small ordeal, and the organizers of the study ran into trouble both getting authors to deposit their own material (the vast majority of authors declined the invitation to self-archive) and getting materials into these systems. What was going into the archive were the raw author manuscripts in PDF form embargoed from anywhere between six months to two years.
The results of the usage analysis showed that publishers did receive more article downloads from their journal websites when manuscripts were freely available from the institutional repositories. But when you look at the data tables, the story is not so clear.
The mean number of publisher downloads recorded from the “visible” group was 17.1 article downloads and 15.3 article downloads from the “hidden” group — a difference of 1.8 article downloads over three months. Not a big difference in practical terms, but a statistically significant one.
However, the mean number of manuscript downloads from the repositories was just 1.77 for the “visible” group and 1.86 for the “hidden” group — a difference of just one-tenth of one download over three months. I don’t really understand how the “hidden” group could have downloads, but if you follow the logic, these very small number of manuscript downloads resulted in 10-times as many downloads to the publishers’ sites.
Now, the authors of the usage study, members of CIBER Research, describe how the metadata deposited along with the article manuscript may be the cause of the increase in downloads at the publishers’ sites, but more research is needed to understand the process — it always is. It could have been the result of indexing robots, which were not excluded from the download counts.
The complexity of the PEER study, and its surprising results, were obviously a source of concern of the authors of the analysis, who discuss its limitations, in detail, much later in the report. They write:
Action research in an environment as complex as the scholarly web is fraught with difficulties and caution must be applied to the findings of the study. We absolutely must not generalise from the findings here to green open access more generally since PEER has a number of characteristics that taken together make it unique. (p.20)
In other words, we should understand this study was so completely unique that the authors find it impossible to make inferences to any form of real-life archiving. The meaning of PEER floats adrift in a sea of ambiguity, unmoored to any plausible future of open access.
Is there nothing that can be learned from the PEER study, other than how to spend millions of dollars and years of publishers’ time? To me, what this study says is that access to final author manuscripts (in their raw form) through institutional repositories with embargoes poses little risk to publishers. Secondly, PEER cannot be generalized to large, systematic archiving of published, formatted articles in central subject repositories, such as the recently published study of PubMed Central.
The reason why this message is lost on readers is because the authors of the PEER study never made it in the first place, and buried the context of their results deep into the report. The great weakness of their report isn’t in the analysis of the data, but in the failure of the authors to frame the research in a clear and coherent message. As a result, open access advocates have snatched this report out the sea and framed in their own terms. Or, as a high ranking STM publisher confided in me last week, “they have taken our study and turned it against us.”
At heart, this argument over archiving has nothing to do with access and everything to do with counting. Publishers want to be recognized and rewarded for the value they add to the scholarly communication process, irrespective of their business model. Those working for PLoS want to see how PLoS articles are being used just as much as any subscription publisher, and for this to happen, the numbers generated by repositories have to be conveyed back to publishers, authors, their funders, and their institutions.
This necessarily increases the administrative burden on repositories; however, if repositories wish to compete as publishers, they have to start operating like one.
Discussion
30 Thoughts on "Institutional Repository Study Is Recast in UK Political Light"
“In other words, we should understand this study was so completely unique that the authors find it impossible to make inferences to any form of real-life archiving. The meaning of PEER floats adrift in a sea of ambiguity, unmoored to any plausible future of open access.”
While I won’t disagree that there’s some truth to this assessment, isn’t it still the case that the PEER results are still the only ones we have? The authors are right to caution about over-interpreting their results; but unless you know different that study is the only game in town. So the alternative to extrapolating from it is to make up results from thin air. (Which indeed seems to be the approach taken by most paywall-based publishers, claiming with no evidence whatsoever that Green OA will harm their business.)
Bottom line: the PEER report provides imperfect evidence, and we’d love to have more and better evidence. But since we don’t, we’re better off leaning on its weak evidence than ignoring it and being completely evidence-free.
“as a high ranking STM publisher confided in me last week, “they have taken our study and turned it against us.””
How very telling that the publisher thought PEER was their study.
“If repositories wish to compete as publishers, they have to start operating like one.”
I would think that most repositories very explicitly do not want to compete as publishers. They want to be the alternative to publishers.
Mike, you’re forgetting the recently published study on the PubMed Central repository which provides results on a natural experiment underway:
Davis PM. 2013. Public accessibility of biomedical articles from PubMed Central reduces journal readership—retrospective cohort analysis. FASEB Journal: http://dx.doi.org/10.1096/fj.13-229922
Thanks, I’d not seen that one. Worth editing the text of this post to mention it? (Not everyone reads the comments.)
The study Phil pointed out is mentioned in the post, which also includes a link to it. It’s in the sentence, “Secondly, PEER cannot be generalized to large, systematic archiving of published, formatted articles in central subject repositories, such as the recently published study of PubMed Central.” (Apparently, not everyone reads the posts.)
Quite right, my bad.
Performing further studies may be easier said than done. Does your university have a repository? If so, does it provide granular, article level metrics on usage that can be parsed by title and publisher? Now ask that question of every university or subject-specific repository out there. The data you are seeking may simply not be accessible at this time.
PubMed Central does provide this sort of information, and the study done there is informative, but only offers limited insights to the overall picture. Since PMC contains only medical and life sciences literature, it tells us nothing about other subject areas, which may have radically different usage behaviors.
Furthermore, since repositories are currently disjointed and contain only a small percentage of published literature, current usage patterns may tell us very little about a situation where repositories are connected, well-indexed and contain the majority of content published. If a reader has to scramble between hundreds of repositories to find a few articles, that’s a very different prospect than finding everything under one roof.
But since we don’t, we’re better off leaning on its weak evidence than ignoring it and being completely evidence-free.
A strange approach coming from a scientist. I was always taught not to draw conclusions at all without sufficient evidence. I hope this approach does not also apply to your academic research.
As a palaeontologist, a lot of the while I deal with samples of size one. It’s better to draw tentative conclusions from little evidence than to throw up your hands in despair. The trick is not to be too wedded to those tentative conclusions, so that when more evidence turns up you’re ready to revise your position.
Tentative conclusions are fine and they are called conjectures. But public policy should not be based on conjectures presented as established facts.
“But public policy should not be based on conjectures presented as established facts.”
Again, I am sympathetic to this. But the reality is that there has to be a public policy. (Continuing with the status quo would be just as much a policy as changing it.) And the alternatives are for that policy to be a based on weakly-evidenced conclusions or on none at all. Seems obvious to me that the former is better. If subsequent studies show different results, the policy will no doubt be revisited.
Guessing, then trying to fix the resulting mess later doesn’t strike me as the best way to run a business or a society. But maybe that’s just me.
For many, the status quo works just fine (see the low compliance rates for most OA policies and the general continued pace of research and technology). The burden of proof is on those seeking change. If the proof that is presented is indeed weak and inconclusive, it should be clearly noted as such. To do otherwise would be dishonest.
Whatever choice our government makes — including the choice to do nothing and leave control in the hands of the historical incumbents — is a choice made on the basis of inadequate data. All they can do is the best they can with what they have. That’s government.
But if they’re stumbling in the dark, they should be made aware that they’re doing so, rather than presenting guesses as certainties.
“The burden of proof is on those seeking change. If the proof that is presented is indeed weak and inconclusive, it should be clearly noted as such.”
The PEER Project was hardly a disinterested party in the case against green open access policies. Publishers were represented. If I recall, the project was initiated in response to emerging green OA policies and to test the impact on those publishers. On that basis, the case not proved here is the conjecture that reduced subscriptions might result from green OA policies. If the evidence on that cannot be found in this report, where is it?
Phil points to the “failure of the authors to frame the research in a clear and coherent message”. That may be so in the final reports, but this can’t be divorced from the original framework for the project on its home page, which is concerned about the “impact the broad and systematic archiving of research outputs in open access repositories” might have on “essential” peer-reviewed journals.
You might accuse supporters of green OA for using this “inconclusive” evidence in support of their case, but equally you cannot blame them for citing the inability of the project to make its own case against green OA policies.
Mike you are confusing policy with action to change policy. Not changing an existing policy is not an action. The basic rule is that if there is found weak evidence that a policy may be wrong then we seek stronger evidence. The evidential threshold for policy change is relatively high because everyone wants something done about something or other. Conjecture does not trigger change (unless it triggers a stampede).
One doesn’t throw ones’ hands up in the air in despair. Instead, it’s perfectly reasonable to propose potential explanations based on initial evidence, but these are far from conclusive and must be stated as such.
When supporting evidence is weak and far from definitive, reputable scientists usually make note of this, and strive to make it clear that anything drawn from the data is at best tentative and should not be considered reliable. Words like “suggested” or “theory” or “possible” are often used. It should be noted that Neylon does not present the PEER study in this light, and instead uses words like “actually” instead.
Correlation is not causation and I see no reason to think that these repository actions caused the publisher activity. Causation requires confirmation of some underlying mechanism. Has any such mechanism been proposed or tested, much less confirmed? Causation always competes with coincidence.
As a minimum we would need to replicate these results in several other trials to even begin to take them seriously. Think of this as an early clinical trial and the unjustified hype makes more sense. Hyping inconclusive early results is a common problem.
This is the kind of comment (and thinking) that separates academics from business people. If I’m running a business, and I notice that another business enters town and starts selling the identical merchandise, and my sales go down 15%, I don’t need to do a deep study to understand there is a threat to my business that is both real and logical. Correlation is enough for me to respond to the business threat. Maybe I’m wrong, but I can test the hypothesis quickly by making myself the exclusive provider of that merchandise in the area. If sales rebound, I solved the business problem. If not, I look for another cause.
The vaunted “progressive workup” is a practical way to tackle real-world situations. Start with the most likely hypothesis, see if addressing that solves the problem, and if not, go to the next one.
The studies suggesting competition from free copies are founded on very logical hypotheses. The best next test from a business standpoint (and perhaps the shortest route to proof) would be to remove the competing copies and see if traffic recovers. If it does, QED. If not, look for other reasons.
Correlation is not causation, but correlation doesn’t mean I should be an apathetic business manager in the face of it.
I agree that the presumption is that free copies hurt business but this PEER study suggests otherwise, that free copies actually help publishers. My point is that the study is too inconclusive to falsfy the presumption, which it is being taken to do by OA advocates. I am merely pointing out what it would take to settle the issue scientifically. Science may be academic but it is not irrelevant.
David, my point in this post was that the PEER and PMC studies are not contradictory at all, but make complete and complimentary sense when viewed within their methods and contexts. Archiving of author manuscripts in institutional repositories with embargoes appears to pose little risk; however, archiving of published articles in XML in a large subject-based repository appears to pose a very large risk.
There are smart people in the advocacy world who would rather count up papers that support or negate a set of findings. I was taught that methodology really matters, and it is sufficient for single studies, if done well, to wipe out entire swaths of the literature. Science is not set up as a democracy. Some studies have much more power in leading scientific discourse than others.
Your first paragraph is an interesting conjecture Phil but I do not think we know that it is true based on the PEER study. I do not understand your second paragraph, except my point is that the PEER study probably has no power at all as is. It is completely inconclusive as a clinical trial.
Hi David,
One feature of the PEER study is that publishers were able to set flexible embargo periods. Green oa done in this way – with flexible, publisher-set embargos – might indeed help everyone. Unfortunately this important ‘detail’ appears to be overlooked too often in green oa policy formation.
I agree Alicia, in fact flexible embargo periods is the third policy monster in my last Kitchen article: http://scholarlykitchen.sspnet.org/2013/02/25/confusions-in-the-ostp-oa-policy-memo-three-monsters-and-a-gorilla/.
The PEER Project has previously tried to deter others’ interpretation of its results http://www.mail-archive.com/goal@eprints.org/msg00600.html
See also Good news for OA mandates: David Prosser’s summary of the PEER study results
https://plus.google.com/109377556796183035206/posts/L2AsGSv4x4W
Your mocking comment about politicians referring to PhD’s as “doctor” detracts from the rest of your fine article. “Doctor” is just an honorific; many don’t care, but it matters to some. I always use it referring to colleagues. Better to overpraise than to insult.
agreed – it is a minor point and it detracted from the relevance of the rest of the article to begin with such an unusual complaint about labels. I guess there may be a difference here between UK and US practice, or even between the quality of PhDs.
It may also be a cultural thing. The only person who ever calls me “Doctor” is my Grandmother.
I get called it all the time but the British are very good at sarcasm.
For the record I am a business consultant not an academic. I walk top executives through complex issues with an emphasis on uncertainty. It is called decision making under uncertainty. I do basic research in the process but my focus is on solving specific business problems.
My comment was a reply to Kent’s comment above not an advertisement. The reply function malfunctioned. See his http://scholarlykitchen.sspnet.org/2013/04/24/peer-repository-study-recast/#comment-94246
I came to this article from a tweet suggesting the PEER study was “misused” at the BISOA hearing. Indeed the article casts suspicion, noting “the context of nuanced studies were stripped away to make simple declarative findings, and…complexity in science was used to make negative categorical statements, like ‘there is absolutely no evidence that…’.”
I watched the entire hearing previously, and just re-watched the section Davis cites, of Cameron Neylon speaking, and tried honestly to see what misuse I might identify. My impression, actually, is that Neylon’s statements seem careful and appropriate, and rather mischaracterized in this article. Here’s what Neylon says:
“we’ve heard a lot of words about consensus there is a problem, we’ve hear a lot about fears…there is very little evidence….the PEER project has been mentioned a couple of times, and usage has been mentioned. The PEER project actually shows there is a small but significant increase in traffic to the publisher’s site.”
You could say that context and complexity were stripped away, but of course they were — he was summarizing a huge study, readily available and already entered as evidence for the committee, in a different context that demanded brevity. That in itself is not misstatement or deception.
He said “there is very little evidence” about repository harm to publishers, not “there is absolutely no evidence that…” as you imply. He then says the PEER study found some increase in publishers traffic; below you say, “publishers did receive more article downloads…when manuscripts were freely available from the institutional repositories.” I’m struggling to see how Neylon said anything essentially different than you do.
Yes, the PEER study authors had some caveats and suggestions about complicating factors like metadata deposit, but surely it’s quite normal for any large social-scientific study to present such notes? They’re qualifications, not invalidations or reversals of the study’s overall conclusion.
The PEER study authors did write “We absolutely must not generalise from the findings here to green open access more generally,” but this is a rather odd statement to take literally. You do a study, it has some strengths or weaknesses, but presumably it offered some evidence of *something*, if it wasn’t malfeasant or totally misguided. Of course it can’t be just naively extrapolated to other cases, but it can be considered as a possibly relevant piece of evidence, caveats acknowledged. As with any piece of research.
Neylon goes on to mention multiple other, seemingly quite relevant other pieces of evidence, such as Davis’ BMC study, several ALPSP publisher surveys, the case of non-scholarly free versions increasing commercial sales, and the case of arXiv, 20 year of zero embargo preprint repository not leading to demise of e.g. physics journals. Why, I wonder, is this range of evidence he brings up not mentioned in Davis’ article?
I’m afraid I have to conclude that Neylon was presenting evidence reasonably and inclusively, while Davis, in this article, is not.
—
Tim McCormick
@tmccormick tjm.org Palo Alto, CA, USA
