Editor’s Note: Today’s post is by Mike Rossner. Mike is a consultant about image manipulation in biomedical research through his company Image Data Integrity, Inc. Mike’s career in publishing included time as the Managing Editor of The Journal of Cell Biology and then as the Executive Director of The Rockefeller University Press.
The STM Integrity Hub
Several posts (“The New STM Integrity Hub”, “Peer Review and Research Integrity: Five Reasons To Be Cheerful”, and “Research Integrity and Reproducibility are Two Aspects of the Same Underlying Issue”) in The Scholarly Kitchen last year described the STM Integrity Hub. This Hub is a platform being developed by STM Solutions, through which participating publishers will share access to their submitted manuscripts. The Hub will include software to detect each of the following: 1) The hallmarks of a manuscript produced by a paper mill; 2) simultaneous submission of a manuscript to multiple journals; 3) image manipulation/duplication. The software applications for the latter two are intended to work at scale, comparing the content of a submitted manuscript to thousands of other submitted manuscripts, and perhaps also to millions of published articles.
Algorithms to detect image manipulation/duplication
My interest in and commitment to image data integrity spans more than two decades. In 2002, I initiated a policy at The Journal of Cell Biology to screen all images in all manuscripts accepted for publication for evidence of image manipulation/duplication. That work was described in a Scholarly Kitchen interview nearly a decade ago. At the time, all of the screening was done using visual inspection, aided by adjustments of brightness and contrast in Photoshop, which can reveal inconsistencies in background that are clues to manipulation, or consistencies that are clues to duplication.
In my opinion, visual inspection remains the gold standard for screening images for manipulation/duplication within an individual article or for image comparisons across a few articles, especially when a processed image in a composed figure can be compared directly to the source data that were acquired in the lab. But that process does not scale to comparisons across the entirety of the biomedical literature.
In the past decade, numerous software applications have been developed for the automated detection of image manipulation/duplication. These applications present the possibility of screening images at a scale that is not practical with visual inspection, and their use has the potential to protect the published literature in ways that were not previously possible. Several of them are now commercially available.
A call for data transparency
A recent news article in Nature indicated that “a small group” of publishers is currently testing the effectiveness of various software offerings in this space on behalf of the STM Integrity Hub. It is important that the data used for those tests, along with the results of those tests, be made publicly available, at least for the software ultimately chosen for the Hub. Ideally, these data include the images that were used for the tests and the output from the software, including the calculated true positive and false positive rates for different types of image data (e.g., photographs, blots, micrographs, scatter plots) and different types of manipulations (e.g., splices, use of the clone tool) or duplications (e.g., direct duplication, duplication with a change in aspect ratio, duplication with a change in orientation). Those rates can then be independently verified.
It is, of course, not unheard of for entities with a vested interest in a product to test it themselves. Think of airplanes being tested by their manufacturers or clinical trials run by the company that produced the drug. However, how crucial to the public good does a product have to be for its validation data to be subject to public oversight, such as the FAA for airplanes or the FDA for drugs in the U.S.?
While the connection to public health and safety may be less direct for much of pre-clinical research, I would argue that integrity of the published record is sufficiently important that the validation data for software designed to protect that record should at least be made public, and at most should be audited by a public entity, such as the Office of Research Integrity in the U.S., or European Network of Research Integrity Offices in the U.K. and Europe.
The importance of data transparency
Anyone using, or considering using, the software selected for the Hub needs to know its capabilities and limitations, so they know to what extent its use is protecting the published record. Any publisher using the software also needs to disclose to its editors/reviewers/readers what the software can and cannot do, so that they can remain vigilant with respect to its limitations. For example, if the software is really good at detecting duplications in certain types of images but not so good at detecting duplications in other types, theeditors/reviewers/readers will know to remain more vigilant about visual inspection of the latter types.
Anyone basing decisions about potential research misconduct on the output of software also needs to know its capabilities and limitations. I just consulted on a case where an author defended himself against an allegation of image duplication by using an algorithm, which did not detect the duplication. The duplication was strikingly evident upon visual inspection of the image, although the author tried to dismiss the visual inspection as subjective.
Visual inspection can take into account pixel variations, such as those introduced by image compression or different exposures, that might fool an algorithm’s statistical analysis. Although the editor of this particular journal did not take the author’s defense at face value, I am concerned that editors will become reliant on software to settle similar matters in the future without fully understanding what it can and cannot do.
I believe that there is a meaningful place for algorithmic screening of image data by publishers before publication ― in conjunction with visual confirmation of the results — but it is important that this community be transparent about the capabilities and limitations of any software that it chooses to use.