This Tuesday, PLoS announced the launch of “The Reproducibility Initiative,” and did so with an admission of something not quite approaching success:

When PLOS ONE launched in 2006, a key objective was to publish those findings that historically did not make it into print: the negative results, the replication studies, the reanalyses of existing datasets. Although everyone knew these studies had value, journals would rarely publish them because they were not seen to be sufficiently important. PLOS ONE sought to become a venue for exactly these types of studies. As it happened, however, the submissions were not hugely forthcoming, although we have published a few.

We’ll get back to the important part of this admission, which has to do with incentives and their relationship to value, a little later.

As announced, the Reproducibility Initiative has a number of factors to it, many of which seem like yet more ways to tap research funds rather than ways to solve a legitimate problem for scientists.

The pitch is that scientists can check whether their experiments are reproducible using an “objective” lab and experts from various institutions, all sworn to secrecy. To do this, participants are paired up with the Science Exchange, which calls itself “The Scientific Services Marketplace.” It seems to work like a contract research organization (CRO), charging scientists to perform certain assays and tests. Prices are listed prominently on the site (an RNA Microassay is $107.50 per sample, for instance), and can be shopped by provider. If you pass the reproducibility test, you get a certificate. The sample on the site made me laugh — it’s a certificate praising Watson and Crick for having validated their methods and results.

The Science Exchange is funded by a group of venture capitalists, another bit of Silicon Valley money injecting itself into the scientific publishing world.

In a nifty bit of sleight of hand, the site claims it’s free with this lovely bit of logic:

No cost for providers to list services ● No cost for researchers to request services ● Easy to pay any provider via payment platform

I love free things that provide easy ways to pay.

The Reproducibility Initiative itself is run on the Science Exchange servers, and uses the Science Exchange domain (https://www.scienceexchange.com/reproducibility) once it redirects from http://www.reproducibilityinitiative.org. It’s a secure site, another sign of its commercial goals.

Essentially, the Reproducibility Initiative seems like a feeder program for Science Exchange providers, who likely provide Science Exchange with a percentage of their revenues as part of the no-cost listing agreement.

PLoS seems to be participating in order to create a feed into its publishing programs, assuring participants that any studies coming from the program will be published:

PLOS will publish all studies validated through the Reproducibility Initiative in a special open access Reproducibility Collection. This will provide you with a second independent publication for your replicated results in PLOS ONE.

“A second independent publication?” Aside from the use of the word “independent,” this sounds like redundant publication, but since novelty or positive results aren’t a pressure PLoS ONE thinks scientists should face, I guess they’re fine with it. Funders might have some issue with paying for experiments and publication twice, however.

There also seems to be a whiff of exaggeration about the level of support the initiative has, with ArsTechnica boasting that the project is “supported by Nature, PLoS, and the Rockefeller University Press.” In fact, the support coming from Nature and RUP is about the weakest kind you can get in publishing — “linking to the PLOS ONE Reproducibility Collection for studies that were originally published in their member publications.”

The Reproducibility Initiative is accepting 40-50 studies to start with, and part of the application process is to divulge the available budget (a required field on the form). Studies “will be selected on the basis of potential clinical impact and the scope of the experiments required.”

Exactly who is screening these submissions isn’t clear. The criteria are loose (“Providers will test for: the reproducibility of your study’s methodology; validation of your study’s results”), and judging by the wording, it seems like a study might be shown to provider after provider until one wants to test it. Not to worry though — it’s all confidential.

There is also a stated belief that “high-profile studies” will be more likely to use this set of services. Most high-profile studies have been reproduced, if they can be; many high-profile studies, especially medical studies involving patient populations, can’t be reproduced exactly, and have to depend on statistical tests to validate their findings. There is no mention of how these studies might be handled, so I think this focus on “high-profile studies” is puffery. The Science Exchange partners seem to focus on basic bench studies and processes, not large-scale studies or highly original techniques.

John Ioannidis, who is quite skeptical about the validity of many studies and who is on the Reproducibility Initiative advisory board, published a list of the problems plaguing studies in a 2005 PLoS Medicine article:

. . . a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.

Basically, little of what Ioannidis articulates can be addressed by the resources Science Exchange can bring to bear on any reproducibility task. In fact, by Ioannidis’ standards, this initiative might add to the problem by creating “more teams . . . in chase of statistical significance” and “greater financial and other interest and prejudice.”

Fundamentally, this seems an odd way to approach the reproducibility problem. One could argue that science itself takes care of the reproducibility problem naturally through the publication of subsequent findings, which either confirm (not interesting) or refute (interesting) published findings.

Paying to reproduce findings after publication in order to get another publication event seems pretty cynical, both on the part of the scientist and on the part of the publisher of either redundant or conflicting results. And the logic of the PLoS part of the process is hard to follow. Let’s say I publish a paper with a decent set of results, strong enough for me to sign my name to the conclusions I propose. Then, for whatever reason, I want to pay another anonymous scientist to reproduce the results after publication. What is my incentive to do this? I have a publication I’ve signed my name to. Either it will be confirmed or refuted. If I feel the results will only be confirmed, what incentive do I have to waste my time and my funder’s money? If I feel the results could or would be refuted, I have a deeper problem with my research claims.

To me, it’s problematic for a publisher to sell access to what it claims are high-quality peer-review and selection criteria on the one hand, then to also sell opportunities to repeat experiments and republish results on the implicit assumption that something different might emerge.

Whatever you think of the ethical dimension, practically this gets back to what I noted at the beginning about incentives. Apparently, the thought behind PLoS ONE was that journals weren’t willing to publish negative results, replication studies, or reanalyses of existing datasets. What they should be learning from their lack of submissions of this type, despite a relatively low bar for publication, is that it’s not journals that weren’t willing to publish these studies — it was scientists who were unwilling to waste valuable time on publishing negative results, replication studies, or reanalyses of existing datasets.

There is also the problem of authorship. Let’s say a 20-author paper is published, and then the authors want to run it through the Reproducibility Initiative. And, let’s say, the findings are so compelling that they require a radically different publication event. What happens to the Reproducibility Initiative workers who discovered the problem — either methodological or technical? Are they now authors on the new paper?

The Reproducibility Initiative takes the inherent barriers to negative, replication, or reanalysis studies and adds further barriers — the cost to do more tests, the potential loss of confidentiality by sharing data with unknown scientists at other institutions, and another cost of publication. How this advances things isn’t clear at all to me.

The Reproducibility Initiative has some mitigating features, the most prominent of which is a strong Advisory Board. Of course, an advisory board’s role doesn’t necessarily mean much in the long run if it’s just window dressing. Governance is probably coming directly from Science Exchange.

I’ve been trying to put my finger on something in all the recent “modern” approaches to scholarly and scientific publishing. Right now, the closest I can come is that it’s all about “reinventing the wheel, but with more jargon and steps.” Alt-metrics measure themselves against the impact factor, and celebrate correlation as validation; “publish then filter” publishing adds a step on top of a filtering step that the approach implicitly elevates but which is more efficiently accomplished by “filter then publish”; and post-publication peer-review is thought useful because, at its best, it helps differentiate research a bit in quality and relevance, two things pre-publication peer-review already does but in a more efficient, controlled, and disciplined manner.

So, it’s not surprising that a new initiative, called the “Reproducibility Initiative,” is another “reinvent the wheel, but with more jargon and steps” approach, with most of the steps aimed at taking more money out of research funds. In this case, what it’s proposing to reinvent is science itself. But this time, with certificates.