Editor’s Note: Today’s post is coauthored by Tim Vines and Ben Kaube. Ben is a cofounder of Cassyni, a platform for helping publishers and institutions create and engage communities of researchers using seminars.
Assessing research integrity issues for articles submitted to an academic journal is a daunting prospect: the journal must somehow use the submitted text, figures, and (if available) accompanying data to uncover unethical behavior by the authors. This behavior (such as fabricating or altering data, etc.) may have taken place long before the article itself was written.
However, these methods primarily focus on detecting bad behavior. To complement this approach, there’s a compelling need for indicators of “honest signaling” in research — a concept that, instead of aiming to catch bad behavior, captures commitment to ethical research practices.
It is instructive to draw analogy to the evolutionary biology concept known as “honest signaling.” This principle illustrates how certain traits in the animal kingdom, such as the peacock’s extravagant tail or the bright colors of some jumping spiders, serve as ‘honest’ markers of the individual’s fitness. These traits are energetically costly to produce and maintain, signifying that only individuals in prime condition can afford these displays. Thus, these signals are a reliable indicator of an individual’s fitness and hence their quality as a mate.
Translating this concept to the realm of research integrity, we seek analogous signals within researcher behavior — practices that are sufficiently demanding of time, effort, or resources, such that they are unlikely to be undertaken by those not genuinely committed to ethical research. Honest signaling in this context would encompass activities that are both visible and verifiable, shedding light on the researcher’s dedication to transparency and integrity:
- Open Science Practices: This includes the sharing of raw data, pre-registration of studies, publishing of preprints, and sharing of the code and scripts to operate command line software. These practices not only require a significant investment of time and resources but also open the researcher’s work to scrutiny and verification by the wider community.
- Community Engagement: Active participation in the scientific community through presentations at conferences, seminars, and poster sessions acts as a form of honest signaling. Such engagements are not only resource-intensive, but also expose one’s research to critique and validation by peers. Moreover, maintaining a visible online presence (e.g., personal webpages and ORCID profiles) and contributions to journal publishing outside of authorship roles (e.g., writing peer reviews and membership on editorial boards) further exemplifies a researcher’s commitment to open discourse and collaboration.
These signals, akin to the peacock’s tail, are not just markers of the quality and credibility of the individual’s work but also serve to signal their commitment to the principles of research integrity.
Any indicators of honest signaling should be considered contextually, acknowledging that open research practices differ between fields and engagement with scholarly communities varies regionally. This is particularly true when considering researchers facing systemic barriers such as those in the Global South. This suggests that simple scores of honest signaling will be overly reductive and that honest signaling indicators should be evaluated qualitatively as part of the editorial process.
The continued growth of digital platforms and the increase in online data post-COVID offer new opportunities to operationalize honest signaling indicators. By aggregating data from society meetings, departmental web pages, and conference programs, it becomes possible to capture engagement of researchers within their respective fields at scale and produce a suite of indicators for research integrity purposes (e.g., what is the publication record of author X? have they presented similar work at relevant conferences or seminar series? Does the manuscript have accompanying data or code?).
One important consideration when aggregating data from many venues is the difficulty of author disambiguation. ORCID has an important role to play here, but the usual challenge of preventing false positives is compounded by the fact that many research outputs associated with honest signaling have traditionally lacked persistent identifiers and high quality metadata. For example posters and seminar recordings have only recently started receiving persistent identifiers.
In biology, honest signals are an indicator of an individual’s fitness, and while some of those signals – like the peacock’s tail – might be eye-catching to us, they are fine tuned to be evaluated by individuals in the same species. Similarly, in the world of research, it will always be the experts in one’s own field that will be best positioned to pick up on and assess honest signaling.
The concept of honest signaling offers a promising framework for verifying research integrity in a manner that complements existing tools aimed at detecting misconduct. By focusing on behaviors that are costly in terms of time and effort and indicate a researcher’s commitment to ethical practices, this approach not only helps identify instances of integrity but also encourages a culture of openness and engagement within the scientific community. Operationalizing indicators of “honest signaling”, while challenging, presents an opportunity to foster transparency and accountability in academic research, ensuring that it remains a pursuit marked by ethical conduct and genuine discovery.
Discussion
18 Thoughts on "Honest Signaling and Research Integrity"
Tim, you provide an interesting thought piece, but I think that you’ve pushed “honest signaling” way past its original meaning and conflated several practical and operational issues of journal publishing into the amorphous concept of “honesty.”
First, the concept of “honest signaling” in biology is contestable, because one needs to ascribe intent behind the action. Is the peacock being “honest” when displaying his tail, or just showing his sexual fitness? And what about all of the examples of mimicry and false signals in the biological world? Is a Viceroy butterfly being “dishonest” by mimicking the pattern of the toxic monarch butterfly so it won’t be a bird’s lunch or is it just being strategic?
Second, you list a number of things that authors can do to increase “honest signals” in publishing, such as depositing datasets, preprints, posters, etc. “Transparency” would be a better, more accurate term here than the amorphous construct of “honesty.” Similarly, we could use the term “longevity” to describe how many papers have been published previously by an author; “peer validation” to describe how colleagues view this author; “merit” to describe the institutions (labs, universities, funders) that vouch for that individual, etc. Honesty is not a yes/no checkbox–I’m not accusing you of writing this–but the logic of your data collection leads an evaluator to make an honesty evaluation.
Like citations or alt-metrics, I fear that your intent on collating these “honesty signals” and using them in the manuscript decision-making process will be counter-productive. Eugene Garfield kept arguing until his death that citations and Impact Factor scores should not be used alone and only in context. Look how far honesty got him.
Comments crossed, but I’m with Phil that running with the ‘honest signaling’ concept from ecology and evolutionary biology doesn’t go far, since there are probably more examples of deception, and deception works. Still, if we think of “Transparency” instead of “honest signaling” I don’t follow the rationale that “collating these ‘honesty signals’ and using them in the manuscript decision-making process will be counter-productive.” Because anything can be gamed and careful fakery is hard to detect? I don’t see that as discounting the value of transparency in the rigor of studies.
The counter-productive comment was about where the argument was heading–creating an “honesty score” or metric that would be used by a triage editor in their decision-making.
Most authors are first authors, so they don’t have any history. And many senior (last) authors would get a very high honesty score on a manuscript but may have little to do the with the integrity of the manuscript, and may not have even read it. Honesty signals just confuses and obfuscates the real details that you want. These can be dealt with by check-listing all the details that you want with a manuscript (data deposit, institutional email address, contribution statement, etc.).
I realize that this cuts out any AI or machine-learning algorithms that a startup company sells to publishers to scour the web, gather and score the “honesty” of any author. Honesty is not an easy construct to operationalize, and personally, I think the outcome of such an effort would be worse than what we currently have. Call me pragmatic.
Hi Phil – you may find this concept easier to digest if you view the ‘individual’ here as the research article, rather than the researcher themselves. The manuscript as submitted is the dull brown bird: it’s very hard to judge whether the authors are just going through the motions to get another publication, or whether they’re committed to contributing to knowledge.
The manuscript equivalent of the ‘peacock’s tail’ are the associated outputs for that piece of research (the datasets, code objects, conference posters, presentations, etc); as these take considerable extra work to produce and also allow readers to inspect the ‘internal state’ of the manuscript. For example, a reader opening the data file and seeing that it’s a disordered mess can conclude that the manuscript is likely flawed. Putting in the effort to make a poster about the research and taking it to a conference shows that the authors want scrutiny for their work.
Just read the manuscript.
Deceptive adaptations are indeed very common in nature – but all the examples you give relate to deceiving other species. Deceiving potential mates about your quality (or lack thereof) is a valid evolutionary strategy*, but this is exactly why mate choice has evolved to focus on traits that are hard to fake. Sure, you’re a good dancer, but first run a marathon & we’ll dance at the finish line.
(*yes, we know evolution doesn’t have intent, but it takes twice as many words to describe it correctly & it’s also boring for the reader)
Nice post. Could adequate methods detail such as presence of RRIDs and a reasonable explanation of the limitations of the study not be a quality signal?
As you know these are nontrivial additions to the paper, that signal quality and integrity. Cell line RRID inclusion is associated with a reduction in the use of problematic cell lines as we showed in 2019 (Babic paper).
Getting RRIDs for all of the reagents & software in the manuscript is a significant amount of work, so authors that put in this extra effort – particularly when the journal doesn’t require it – should get some recognition.
Excellent essay. I must admit I liked it in part because it fits my view that research integrity and reliability is enhanced through greater transparency, not by hiding honest signaling or the lack thereof behind double-blind reviewing. Your essay seems to target editorial offices to look for honest signaling, but I think a large part also falls to the peer reviewers. The argument for blinding reviewers to signaling, honest or otherwise, is that blindness prevents status bias wherein manuscripts from Prof. Big Name from Elite University are welcomed in the Journal of High Impact Factors where author No Name from Southern Latitude University is not. I’m not sure what the best antidote is for the status bias issue, but I don’t think hiding honest signaling from reviewers is it.
I know this is meant well but it reads like a protocol for being a good Soviet citizen to me. I wonder how many Nobel prizes were won by good boys and girls. In a wider publishing context it comes across like not publishing JK Rowling for her personal views. We are publishing research articles not judging the authors
Strongly agree. The data in each article need to be evaluated for their merits and integrity. There are no work arounds.
The authors don’t seem to be arguing that papers lacking these signals shouldn’t be published. But it is reasonable that readers should view a research article in which all the data are online differently from one in which not a scrap of data was shared. The data is the evidence – it’s what makes science reproducible and falsifiable. Nullius in verba, right? Similarly, a researcher who issues a pre-registration is holding themselves accountable to a stated hypothesis and research plan. One who does not is free to move the goal posts surreptitiously as many times as they want. The difference here really matters in science, and we should notice it.
Great article. I love your perspective. I think you’ve (re)coined a powerful concept for Research Integrity. You draw your parallels from biology, while I draw my parallels from the financial industry.
Online credit card fraud has similar integrity issues. They are solving these issues with Know Your Customer (KYC) controls and “honesty signals” as to whether an individual transaction is genuine or fraudulent.
As you suggest, I don’t believe you can fight AI with AI, in some kind of AI arm’s race. If we’ve learnt anything from the Ancient Greeks, it’s that if you want to beat an opponent one most focus on the Achilles heel. In the case of paper mills, bad actors and fabricated results, that’s truth and reality… and as you have so eloquently put it… honesty signals.
Thanks for this interesting piece, Tim and Ben. I am not sure the concept of “honest signaling” is a perfect analogy for assessing the integrity of a researcher or research output, particularly if there is an implication that additional work might need to be performed to create these signals beyond the good-faith work that honest researchers undertake anyway. To me, this has similarities with the wasteful effort that is caused by the “proof of work” algorithms used by some block chain algorithms.
However, I am not sure that you are suggesting this. Perhaps, you are instead proposing that the work that honest researchers do anyway through, for example, following open science practices and engaging fully in their communities, can act as positive signals of their integrity if shared transparently, accessibly and in a verifiable way. This aligns very much with the work that we are doing at ORCID to encourage the sharing and re-use of what we term “trust markers”, those elements of an ORCID record that constitute positive signals of a researcher’s contributions over time.
In addition to the core elements of publications and other research outputs, education and affiliations, and funding awards, it is already possible to include acts of professional service in ORCID profiles. Such activities can include participation in boards and committees, membership of standards bodies and expert panels, the organization of and participation in conferences, and of course membership of editorial boards. Like all data in ORCID records, subject to the privacy controls of the researcher, this data is available for inspection and re-use by anyone who might find it useful, and always includes provenance information (i.e. who made the assertion and when).
To make it as easy as possible for researchers to share this information about themselves, we encourage the conveners of such activities, who are the authorities on who is participating, to contribute that data directly to ORCID (always with the researcher’s permission). By doing so, organizations join what we call the Community Trust Network, increasing the overall transparency of research activities, allowing maximum re-use of this information, all while without unduly burdening the researcher with additional work to maintain and share this data manually.
Hi Chris – we’re definitely focused on the activities that diligent researchers do as part of conducting and communicating their work. It’s great that ORCID is working to systematically collect and expose these.
Kudos to the authors — this was really well articulated and the analogies were cute and helpful! [start shameless plug] It’s also a great complement to this SKitch article right here: https://scholarlykitchen.sspnet.org/2021/06/02/guest-post-the-10000-watt-bulb-how-preprints-shine-a-light-on-misconduct/ [end shameless plug].
We know instinctively that transparency and deception are anticorrelated, but the dominant research communication processes and incentive systems favor opacity.
Call them “honest markers”, “trust signals”, whatever…we need more of them and more incentives for their use. Happily, most of them have the added benefit of making the research more reproducible!
Quibbles about evolutionary biology aside, this article makes an important point. Naively trusting submissions because of words and graphs on a page, or even an author name, leads to negative outcomes, especially as human- and machine-agent productions are becoming more difficult to distinguish on a daily basis. Hard to fake, multimodal, and transparent open science practices are an important part of the emerging discipline of forensic scientometrics, and publishers need to become adept at this quickly if they are to remain relevant in the rapidly evolving open science ecosystems.
Conceptually sound, but it’s strange (and more than a little sad to me) that preprints have been adopted here as a part of “open science” as though they’re an unmitigated good. They’re hardly in the same league as registration of prospective trials or data sharing. Neither of those has caused harm, whereas preprints certainly have, at least in biomedical publishing. I would urge deeper consideration on that point. (And yes, I’m well aware of the Gates Foundation’s recent moves in this area, which I see as a great example of a self-interested play, given GF’s connection to F1000). –SSL
