Failing to observe a platypus laying eggs is not a demonstration that the platypus does not lay eggs. You have to actually observe the provenance, ab ovo, of the little newborn platypusses, if you want to demonstrate that they are not engendered by egg-laying.

Failing to observe a significant OA citation Advantage after a year (or a year and a half — or longer, as the case may be) with randomized OA does not demonstrate that the many studies that do observe a significant OA citation Advantage with NONrandomized OA are simply reporting self-selection artifacts (i.e., providing OA selectively for the more highly citable articles.)

You first have to replicate the OA citation Advantage with NONrandomized OA (on the same or comparable sample) and then demonstrate that randomized OA (on the same or comparable sample) eliminates the OA citation Advantage (on the same or comparable sample).

Otherwise, you are simply comparing apples and oranges (or eggs and expectations, as the case may be) in reporting the failure to observe a significant OA citation Advantage in a one one-year (or 1.5 year) sample with randomized OA — along with the failure to observed a significant OA citation Advantage for nonrandomized OA for the same sample either (because the nonrandomized OA subsample was too small):

The many reports of the nonrandomized OA Citation Advantage are based on samples that were sufficiently large, and based on a sufficiently long time-scale (almost never as short as a year) to detect a significant OA Citation Advantage.

A failure to observe a significant effect with small samples of short time-scales — whether randomized or nonrandomized — is simple that: a failure to observe a significant effect: Keep testing till the size and duration of your sample of randomized and nonrandomized OA is big enough to test your self-selection hypothesis (i.e., comparable with the other studies that have detected the effect).

Meanwhile, note that (as other studies have likewise reported), although a year is too short to observe a significant OA CITATION Advantage, it was long enough to observe a significant OA DOWNLOAD Advantage — and other studies have also reported that early download advantages correlate significantly with later significant citation advantages.

Just as mating more is likely to lead to more progeny for platypusses (by whatever route) than mating less, so accessing and downloading more is likely to lead to more citations than access and downloading less.

I recognize that there was a tradeoff in choosing to submit our paper for publication with only one year of post publication citation data. We decided it was worthwhile to report preliminary results, rather than wait for more citation data, because of the importance of the issue and because of the stark contrast between our results and those in prior studies. Based on the effect size reported in previous studies, and our statistical power, we should have seen a significant open access effect by the end of the first year.

To further assess concern of insufficient timeframe, we have gone back and reexamined the issue with additional months of citation data. Since our manuscript was submitted to BMJ (with citation data from 2 January, 2008), we have run several update analyses.

As of 3 August, 2008 (15 to 18 months after article publication) the effect of randomized open access on citations remains insignificant (Incident Rate Ratio = 1.07, 95% confidence interval 0.95 to 1.20, P=0.23). Open access and subscription-access articles both have an average of 3.8 citations.

In sum, we still find no open access effect on citations. Nonetheless, we plan to gather more citation data for these two sets of articles, and reexamine this issue again, after allowing even more time to pass.

Self-archiving control
Professor Harnad comments that we should have implemented a self-selection control in our study. Although this is an excellent idea, it was not possible for us to do so because, at the time of our randomization, the publisher did not permit author-sponsored open access publishing in our experimental journals. Nonetheless, self-archiving, the type of open access Prof. Harnad often refers to, is accounted for in our regression model (see Tables 2 and 3).

We could identify only 20 instances of self-archiving, 11 cover stories, and 4 press-releases. Without doubt we lack the statistical power to report an effect with sufficient certainty and do not draw our conclusions based on these limited data. These variables were not the main variable being tested in our study (it was randomized open access).

To summarize, we believe that our research provides strong evidence that open access increases the dissemination of scientific articles, as indicated by our download results. However, we find no evidence of an open access citation effect, even after incorporating six additional months of citation data.

There are many societal benefits to making the scientific literature freely available beyond the research community; a citation advantage may not be one of them.

Excerpt from:

Author’s Rapid Response on BMJ.

It might also be a bit simplistic to refer to the PLoS study as a study “asserting a citation advantage”. If one reads the study in full rather than just skimming through the abstract (which few people seem to do) one would notice that statistically adjusting for all known confounders actually led to an elimination of any citation advantage of self-archiving (green OA) – the citation advantage only remained for gold-OA. Gold-OA articles have a citation advantage of 25-40% (which is already less than what previous quick & dirty studies have asserted – Harnard to this date disputes the presence of any bias or confounders in his studies which talk about 200-700% citation advantages). This advantage holds if we adjust for known confounders.

No-one disputes that an RCT is a superior methodology (I am also doing one), however, the main critique of this particular RCT is the ridiculously short follow-up period and that several “control” variables which SHOULD have been predictive for a citation advantage (articles with press-release, articles on the cover page, self-archived articles) are not significant predictors for citations either.

I disagree with Harnard that there was “no self-selection control condition” in the Davis study. The self-selection control condition is “self-archiving”, which – in line with Davis’ own argument – tend to be the “better” studies, which is why previous studies simply comparing citations of self-archived articles vs non-self-archived articles without adjusting for anything did see a citation advantage of self-archived studies. But the fact that “self-archiving” is NOT a predictor in the Davis study is a major paradox which only leaves the conclusion that the study is not internally valid. Davis should have waited with his publication long enough until all the other variables which are expected predictors for citations become significant.

For a full critique and questions for the author see also
http://gunther-eysenbach.blogspot.com/2008/07/phil-davis-open-access-publishing.html

“Davis et al’s 1-year Study of Self-Selection Bias: No Self-Archiving Control, No OA Effect, No Conclusion”
http://openaccess.eprints.org/index.php?/archives/441-guid.html

Summary:

(1) No self-selection control condition, showing the OA citation advantage, hence no evidence the null effect is a result of eliminating the self-selection citation advantage.

(2) Most studies reporting an OA citation advantage report failing to find the effect in the first year: the time-base is too short.