In 2018, I talked with Melinda Baldwin (Associate Professor of History at the University of Maryland) about a fascinating article she authored entitled “Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States” (Isis, volume 109, number 3, September 2018).
In talking again with Melinda this month, we initiated a discussion of whether peer review has a role to play in uncovering scientific fraud. Perhaps in the first blush, one may suggest that peer review is entirely appropriate as a fraud detector, but Melinda suggests this is not the role of peer review
I ask Melinda here to take us through her perspective on the role of peer review in context for scientific fraud from her perspective as a historian of science – a perspective that I hope you agree sheds a clear light on the benefits and limitations of peer review in the scientific endeavor.
What is the role of peer review?
I’m going to give an annoying non-answer: it depends on who you ask! Journal editors may have different answers from scientists, who have different answers from members of the public interested in scientific findings.
One thing that almost everyone agrees on, however, is that peer review is supposed to be a gatekeeper in the world of scientific publishing. The process of peer review is supposed to detect bad science and prevent it from getting into print. You see this vision of peer review in some of the recent coverage of Elizabeth Holmes’s trial in California. Nature ran a news article suggesting that if Holmes’s company, Theranos, had been expected to publish more peer-reviewed papers, their fraudulent practices would have been detected far earlier.
But, as Scholarly Kitchen readers probably know, being peer-reviewed doesn’t ensure that a scientific paper got it right. There are a lot of cases where papers do make it past peer review and are later found to be deeply flawed — or, worse, are found to be based on deliberately fabricated data.
Is there a role for peer review in uncovering fraud?
I certainly think it’s possible for fraud to be uncovered during the peer review process. A careful referee may notice inconsistencies in the methods or data that suggest the authors aren’t being honest about their results.
But as a general rule I think peer review is not set up to detect fraud, and we shouldn’t be surprised when fabricated data makes it through a peer review process. Most scientists approach the peer review process with the basic assumption that the authors are being truthful when they describe their setup and results. If a referee starts with that collegial assumption of honesty, they’re probably not going to go through a paper’s data with a fine-tooth comb looking for clues that it might be falsified. One notable recent case of a peer-reviewed paper that faced fraud accusations was the “When Contact Changes Minds” paper in Science by political scientists Michael Lacour and Donald Green. The paper passed Science’s peer review process—but it wasn’t until another political science graduate student tried to replicate Lacour’s survey methods that anyone realized something wasn’t right.
Furthermore, a referee who openly accused authors of fraud or misrepresentation would be getting themselves into a potentially thorny situation. Do they include that accusation in a referee report? Have a quiet word with the editor about their suspicions? Recommend rejection and say they’re not convinced by the conclusions, but without saying outright that they think the work is being misrepresented? Even though referees’ names are usually not shared with the author, the journal’s editors will certainly know who wrote the report, and I think many scientists — especially junior ones — might worry that they will come across as petty or malicious if they speak up about their suspicions. The Committee on Publication Ethics (COPE) offers some suggestions for how referees and editors might approach potential cases of fraud.
From your position as a historian of peer review, are there examples of academic fraud and how they were uncovered?
Generally speaking, when fraud is uncovered, it’s uncovered not by the peer review process, but after publication. The Lacour and Green paper is one example. In the case of Theranos, a paper in a journal called Hematology Reports attracted some post-publication scrutiny that contributed to the company’s collapse. A pathologist named Adam Clapper looked up the Hematology Reports article and wrote a blog entry saying that he didn’t buy the company’s claims about its technology. After looking into Theranos more, Clapper became so suspicious that he contacted Wall Street Journal reporter John Carreyrou, whose reporting ultimately brought down the company — see chapter 18 of Carreyrou’s book Bad Blood for more on that.
There are also a host of examples that occupy a grey area — they’re arguably not quite fraud, because the scientists behind these papers seem to genuinely believe what they’re claiming, but they’re studies with serious problems that end up not being replicable. With those papers, too, we see that when the problems come to light, it’s almost always because of post-publication scrutiny. One historical example is the French physicist René Blondlot, who claimed to have discovered something he called an “N-ray” in the early 20th century. When researchers were unable to replicate his published results, an American physicist named Robert Wood traveled to Blondlot’s laboratory to see Blondlot’s procedures. Wood found that if he removed what was supposed to be a key piece of Blondlot’s apparatus — a crystal that was meant to refract n-rays onto the detector — Blondlot reported no change in his results. So that’s an example of a deeply flawed study whose flaws were uncovered by post-publication investigation.
A more recent example, and quite a famous one, is that of cold fusion. University of Utah chemists Stanley Pons and Martin Fleischmann claimed they could generate room-temperature nuclear fusion using a palladium cathode, and secured publication for their work in the Journal of Electroanalytical Chemistry. But when other laboratories were not able to replicate the results they described, the findings were largely discredited.
Are there example of peer review entangling with issues of fraud, successfully, or unsuccessfully?
There have been several examples of fraud in the peer review process itself! In 2012, there was a famous case of a researcher who entered false email addresses for suggested reviewers. The researcher then accepted the invitations to review the paper and submitted favorable referee reports for his own work. It happened again in 2017 at the publisher Frontiers, and in 2019 with nine papers at Elsevier journals. More publishers and editors are now taking steps to verify reviewer identity, but I think it shows that the peer review process itself can unfortunately be vulnerable to bad actors.
But I can’t think of any documented examples of fraud being uncovered during the peer review process. It’s possible — likely, even — that such a detection has happened, but that it’s been handled confidentially and not publicized. I know in history, there are a few anecdotes that make the rounds during conferences about plagiarized papers being withdrawn after a referee points out the plagiarism. I’d be very curious to do an anonymous survey of scientific journal editors about their experiences with any academic misconduct they’ve uncovered prior to publication!
What potential solutions do you recommend for uncovering academic fraud?
I think one key component of any solution would have to be protection for whistleblowers. David Broockman, the graduate student whose analysis revealed the irregularities in the Lacour and Green paper, has said that most of his mentors and colleagues discouraged him from looking so closely at the data behind “When Contact Changes Minds.” They were afraid this young scholar was going to build a reputation as a troublemaker and have difficulty getting a job. If it’s likely that raising questions about a paper is going to damage the questioner’s career, we should expect that most people will try to ignore their suspicions and move on — especially given how competitive the academic job market is these days.
But too much support for self-described whistleblowers could also lead down some dark paths. For example, in the 1980s, there was a very thorny episode involving accusations of research misconduct against a biologist named Thereza Imanishi-Kari. Margot O’Toole, a postdoc in Imanishi-Kari’s lab, was not able to replicate the results from a paper her supervisor had published. O’Toole eventually accused Imanishi-Kari of misrepresenting her data. Because that study had received NIH funding, Congress got involved, the Secret Service went through Imanishi-Kari’s laboratory notebooks, one of Imanishi-Kari’s coauthors resigned a position as a University president — it’s one of the most dramatic cases of fraud accusations in modern science.
Ultimately, Imanishi-Kari was exonerated of deliberate wrongdoing, but not before a lot of collateral damage had occurred. And one of the things that I think makes this such a fascinating and troubling example is that both Imanishi-Kari and O’Toole were arguably acting in good faith — they both genuinely believed they were in the right. I think a lot of investigations into suspected fraud may end up in a similar murky area, where perhaps a study wasn’t done as rigorously as it could have been, but it’s not clear that there was a genuine intent to deceive. So I think one important step will be to think about where an experimenter crosses the line from being merely sloppy into actively committing academic fraud.