A recent study by John J. Regazzi and Selenay Aytac published in Learned Publishing proposes to discern the attributes authors most closely associate with journal quality.

Now, I wish I could write flattering things about this study. Instead, I’ve been worried that I should just stifle myself lest I be accused of being a meanie. As you’ll see, reputation matters, and I don’t want the reputation of being mean. So, let me take you through this gradually so you can see that I’m not being mean, just dealing with a study that left me underwhelmed and a little frustrated.

The authors use a “triangulation” approach (three different research methods) to discern journal quality measures that matter to researchers. They admit they had constraints of “time, cost, and sample size.”

The authors sampled a total of 13 people (7 from a computer sciences discipline, 6 from health) from their own institution (Long Island University). This was a convenience sample, as the research euphemism goes. These 13 people were surveyed, then took part in a focus group, and then, if they hadn’t had enough, they were interviewed. Only 5/13 stayed on for the interviews.

So, 13 people were asked to answer written questions, then asked similar questions in a focus group, and then ~1/3 of them stayed on for interviews.

The authors list 16 quality characteristics they claim to have found in the literature. Looking over this list, you start to see how this is a conflated set of characteristics — for instance, reputation, impact factor, rejection rate, editorial board, and society-published are prone to skew to the most general of the set (reputation) since the others are specific relative to the proxy. So, in fact, you have one summarizing characteristic (reputation) that suffices to encompass the others. It’s bound to score higher.

Then, on what appears to me to be a small and localized data set (thrice-processed opinions — “triangulation” turns out to mean asking the same people something three times), I find the authors rolling out 5 tables, 2 figures, and 7.5 pages of discussion. On top of this, they estimate p-values and invoke Cronbach’s alpha. I thought this was all a bit much, especially in a qualitative study.

Not everything we study has to be subjected to the scientific method, or some imitation of it. Sometimes, good old common sense works just fine.

And that’s what ultimately comes from this small, local study of faculty in computer sciences and health — researchers want to publish in journals with great reputations that get their papers out quickly and reach a lot of readers. Computer science people don’t care as much about online submission tools, but health researchers want them.

So, back to the theme of reputation. Why did I read this article? It was forwarded to me in an email, and when I saw it was from Learned Publishing, I figured it had to be good. Well, even they don’t bat 1.000, I guess. The person who forwarded it to me has a great reputation, but this particular article wasn’t to my liking. Both reputations are well-preserved. I still think Learned Publishing is a great journal, and that my colleague/friend has great taste.

Reputation is transmissible. Phil Davis mentions the concept of “brand coat-tails” in a recent post, showing how PLoS has used two good journals’ reputations to extend acceptance of a repository journal.

Because reputation is transmissible, researchers want to publish with high-reputation journals. They want some of that reputation for themselves. It’s a double-edge concept — reputation becomes akin to a standard.

Phil Davis (once again) suggests something along these lines in a comment on an earlier post on this blog. Phil notes a 1978 study that showed how the act of citation is akin to communal signaling. I would stretch this a little more by stating that reputation is a different form and culmination of communal signaling. By publishing in Learned Publishing, these authors signaled their audience and research interests. And by publishing this paper, Learned Publishing said something about a topic it has consistent interest in exploring. So, despite this being a small study with a conveniece sample and overheated interpretation, it fits for both the researchers and the journal.

It sends the right communal signals, reputation-wise.