Last week, Stephen Colbert interviewed Leon Wieseltier, editor of the New Republic. Ever the provocateur, Colbert immediately challenged Wieseltier to state his critique of modern culture in 10 words or less. This is what Wieseltier came up with on the spot:
Too much digital, not enough critical thinking, more physical reality.
Ten words exactly. Colbert was duly impressed.
I encountered this critique a day after reading a number of articles with issues of data access (open data) and data reanalysis at their core. One of these also touches on the seemingly endless attempts to link vaccines and autism, despite repeated results showing no link, including said reanalysis itself. Overall, early signs indicate that there are significant responsibilities coming our way as we make data more accessible, and we will need to carefully consider how to proceed.
In August, the BioMed Central journal Translational Neurodegeneration published a paper by engineer-turned-biologist Brian Hooker, in which Hooker reanalyzed 2004 CDC data and found a purported link between measles-mumps-rubella (MMR) vaccines and autism diagnoses. His reanalysis showed no link, except in a subgroup of African-American children. Unfortunately, it was a case-control study, and he reanalyzed it as a cohort analysis, just the beginning of the flaws in the paper.
The story that has emerged since then is long and sordid, involving a strange and sensationalistic video of a supposed confession of data suppression (by a “senior CDC researcher” with no trace of existence on Google and other factual problems), an admission from Hooker that he is not an unbiased participant due to his own son’s autism, a panicked takedown of the paper by the publisher shortly after publication, and the reappearance of Andrew Wakefield as narrator of the purported video confession. You may recall that Wakefield is the British physician whose paper started all this, and who was subsequently disgraced when the paper was found to be fraudulent and was retracted.
Last week, the Hooker paper in Translational Neurodegeneration was retracted, with this portentous statement:
This article has been removed from the public domain because of serious concerns about the validity of its conclusions. The journal and publisher believe that its continued availability may not be in the public interest. Definitive editorial action will be pending further investigation.
Why did this even happen? Data reanalysis sounds perfectly benign but belies serious motivation. As one astute blogger writing on Respectful Insolence writes about this instance in particular but reanalyses in general:
There are a couple of things you have to remember whenever looking at a study that is billed as a “reanalysis” of an existing data set that’s already been published. The first is that no one—I mean no one—”reanalyzes” such a dataset unless he has an ax to grind and disagrees with the results of the original analysis so strongly that he is willing to go through the trouble of getting institutional review board (IRB) approval, as Hooker did from Simpson University, going to the CDC to get this dataset, and then analyzing it.
This example of reanalysis shows some of the serious risks associated with an imprudent embrace of open data initiatives — the ability of a motivated individual or activist group to create “findings” that suit their worldviews and publish these to compete with actual firsthand science; the difficulty of teasing out post-hoc statistical manipulations; the lack of incentives to monitor these reanalyses and the burden they may create for editors, publishers, and scientists; and the potential for commercial or reputational or political gains to be had via even imperfect yet confusing reanalyses.
An editorial in JAMA published last year addresses these and other concerns eloquently, stating:
The value of reanalysis . . . hinges critically on reducing the presumed threats to equipoise that come from financial, ideological, or political interest in the results. . . . if the presumption of bias is higher in the reanalysis team, data sharing will likely impede, not improve, scientific understanding.
A recent study, also from JAMA, looked at reanalyses in medical journals and makes what appears to be a significant point — that 35% of medical studies receiving independent reanalysis had their conclusions modified to either change the treatment group, the treatment options, or both — but there is more to it than the surface findings. This article, once again from the prolific John Ioaniddis and collaborators, revealed, but did not draw adequate attention to the fact, that the majority of the studies reanalyzed initially had null findings, while the reanalyses generally increased interventions. Of the 13 reanalyzed studies that led to changes in interpretation, nine (69%) suggested that more patients should be treated, while only one (8%) showed that fewer patients should be treated. In many cases, newer drugs were suggested. One reanalysis suggested that younger patients should be treated, which would result in more years for a costly intervention.
Essentially, if these reanalyses were heeded, patients would be told, “The original study said not to do this, but other people looked at the data again and think you should receive this treatment. Hope that’s OK.” Meanwhile, the financial upside of these reanalyses is potentially significant, while the possibility to generate new, positive results from reanalysis brings us back to the concern overall that scientists want to find positive results because they are more publishable. In the JAMA study, we see many null results being transformed into positive results via reanalysis.
Investors could also benefit, because the stakes in medical research are so high. A recent in-depth story in the New Yorker revealed that one investment firm was able to use inside information around a potential Alzheimer’s drug to generate a $275 million investment gain by selling short the drug-maker’s stock. The investment firm had held millions in stock just weeks before a major study of the drug was announced, but quietly reversed its market position after one of its investment managers, who had befriended the person set to announce the trial results, learned that the results were underwhelming and determined they should buy against their previous position.
These kinds of incentives could lead to many reanalyses, especially if investors aren’t able to anticipate negative results. With hundreds of millions at stake, why not hire some statisticians to see if they can eke out a positive interventional effect? Or at least confuse the markets long enough to recoup your investment?
The New Yorker story also reveals a troubling fact — many medical researchers are susceptible to the charms of temporary business friends. In this case, the senior researcher thought that an investment manager shared his passion for Alzheimer research. Over months, the families became friendly and the researcher held dozens of private consultations with the investment manager. The researcher later admitted to being unable to detect exactly when he’d crossed the line. Once the drug’s poor results were announced, the investment manager stopped emailing this senior academic and stopped replying to messages.
This underscores the difficulty of finding “unbiased” individuals to reanalyze study results. Bias is subjective and manageable — that is, skilled investment managers, reporters, company officials, or academic leaders can wend their way into a position of influence or awareness without triggering alarms. They can even affect the thinking of scientists without their being sufficiently aware of it, through simple and innocuous-seeming persuasion. Thinking is malleable. Permissions are subtle. Some people are gifted at modifying thinking and permissions without leaving a trail.
Disclosures are viewed as a tonic for bias, as they at least make others aware of the potential for bias. But there are many problems with this theory, as a recent Nature news article discussed, including incomplete data, mixed reception by patients, and industry’s ability to find ways into relationships that don’t require disclosure.
Aside from the potential for conscious or unconscious bias, questions around the data used in reanalyses are not trivial. What comprises a complete set while preserving patient confidentiality (not established with a reanalysis group)? Are the data the only aspect that require reanalysis, or do the statistical outputs and methods also require reanalysis? Who provides those? And who reanalyzes the reanalysis, or is a reanalysis the last word?
There is a very good example of competent data reanalysis currently underway, and that is around the BICEP-2 findings which suggested researchers had found the gravitational wave signature of the Big Bang. Concerns were immediately raised about estimates of cosmic dust, and whether misestimating these levels could have led to spurious conclusions. Researchers from the US and Europe have now joined forces, shared data, and will issue a joint reanalysis by the end of the year. Already, the findings have caused the original authors to lower their confidence levels.
It’s worth noting that astronomers and astrophysicists are essentially immersed in data all the time. Data-sharing is integral to their work, and they have a culture and history that fosters such work. This kind of collaboration is important, as a data set may contain unspoken or unknown problems, which become truly unknowable when only the data are analyzed and the researchers excluded. David Crotty has written eloquently about the little perturbations an experimental environment or study design can experience, disturbances that can make reproducibility impossible, confuse researchers, or scuttle results. Reanalyses use statistical tests on data that may contain unknowable problems, carrying them forward in some way. In contrast, another trial of the same hypothesis would likely not have the same problems — it may have new ones, but if it generally arrives at the same outcome, you can have more confidence in the reality of the matter.
Ultimately, the questions for editors and journal publishers are complex and not easily answered:
- What are the authorship criteria for reanalyses? What kinds of disclosures must be made?
- How much validity are reanalyses given?
- How do you peer review a reanalysis? Do you have to re-review the original study as well?
There is a seductive thread in the midst of all this, the idea that we can reprocess reality until we get an answer we like. Pulling on this thread introduces the question of when it stops. Do you reanalyze the reanalysis? How many “re-s” are allowable? More importantly, if reanalysis is embraced as another form of publication, what opportunity costs will this introduce, as more scientists, statisticians, and analysts turn their attention away from actual studies to reanalysis? As the JAMA editorial from 2013 states it:
. . . because the universe of researchers with the expertise, time, and interest in reanalyzing another researcher’s data can be quite small, there is a worrisome likelihood that those seeking to reanalyze data either have a vested interest in refuting the findings or do not have adequate methodological expertise. Because methodology drives the quality of results, this is not an idle concern. Anyone sufficiently motivated can produce a different and conflicting result based on data that once demonstrated a given outcome. A misspecified model, a reduction in power, inappropriate controls; the pitfalls are many and one investigator’s pitfall is another’s opportunity.
What really needs reanalysis is the notion that reanalyzing packaged trial data is a simple matter of access and nothing more. There are significant issues to understand and sort out before we embrace these approaches fully. One main question is whether reanalysis stacks up well to simply restudying the hypothesis. To return to the problems diagnosed by the editor of the New Republic, is science going to be an area with the same deficits as our broader culture:
Too much digital, not enough critical thinking, more physical reality?
Discussion
8 Thoughts on "Virtual Reality Research — Some Early Problems with Data Reanalysis and Risks of Open Data"
The climate change debate may be an extreme case of this problem, in part because there is a limited amount of data and it is subject to many diverse analyses. For example, consider a simple concept like the global average temperature over time. There are four different groups that estimate this based on various subsets of the historical thermometer data, and using different statistical methods. Then two other groups estimate it from satellite data begining in 1978, again using different methods. All the results vary from group to group. So what the data says is far from clear. It gets worse when other data is considered, hence the confusion.
We also see the US Congress actively re-analyzing NSF grants even before they are published. From a recent Chronicle of Higher Education article by Paul Baskin:
Two years into the latest round of attacks by Congressional Republicans on federally sponsored research, an escalating effort by the House science committee to find fault with the National Science Foundation is taking a growing toll on researchers.
NSF grants to some 50 professors across the country are now being investigated by the Republican-controlled committee. More than a dozen of the researchers, in comments to The Chronicle, said they had little idea what the politicians were seeking, but warned of a dangerous precedent in what they described as a witch hunt.
I do not think that the Congressional staffers are reanalyzing data in the sense that Kent is discussing. My understanding is that they are looking for mostly political science research that they deem to be pro-Democratic. For example, surveys into how much some people dislike Republicans (which of course is a lot). The FIRST bill actually has language designed to curb this practice, in a round about way.
But I have not followed the issue closely, so maybe there is more to it than this. If the staffers actually have people re-crunching the numbers that would be interesting indeed. Is there any report of actual re-analyzing of data?
Here is some recent news on this controversy, which may be Constitutional in scope. No indication that re-analysis is involved.
See http://www.aip.org/fyi/2014/controversy-over-nsf-grants-continues
See also http://scholarlykitchen.sspnet.org/2013/03/26/federal-research-funding-and-the-unwillingness-to-cut-bait/.
Congress has a long history of nitpicking research grants. It was Sen. Proxmire who had a rubber chicken award.
Perhaps an interesting example worth considering here:
http://www.the-scientist.com/?articles.view/articleNo/41239/title/Epigenetics-Paper-Raises-Questions/
I am concerned about the requirement of the PLOS journal fleet to make all data sets available, e.g., as open access Excel files. The idea in principle is excellent, and should increase transparency. But, very practically, it could also lead to data theft followed by data manipulation. I am not sure if the “open data” protagonists have even considered this serious and real threat. Without trying to insinuate any class profiling, try to imagine a scientist from developing country A who picks up an Excel file from the PLOS web-page published by a scientist from developed country B, and then simply manipulates the raw data to generate a new set of brilliant data that could result, almost miraculously, with sufficient writing skills, in a paper good enough for a high level journal. The thought is terrifying, even more so because there is abslutely no way of knowing if this is already taking place. The term scientific terrorism, a neoligism I introduce today, is on evolution’s doorstep and the aggressive stance by those who are advocating (correctly so, unfortunately) the open data movement are failing to see the darker side, which has not yet been discussed, explored, or measured.