Comments on: Gaming the Rating System http://scholarlykitchen.sspnet.org/2009/08/05/gaming-the-rating-system/ What's Hot & What's Cooking in Scholarly Publishing - from the Society for Scholarly Publishing Thu, 16 Feb 2012 03:31:36 +0000 hourly 1 http://wordpress.com/ By: An Inflection Point for American Public Libraries | In the Library with the Lead Pipe http://scholarlykitchen.sspnet.org/2009/08/05/gaming-the-rating-system/#comment-4863 Wed, 30 Sep 2009 11:03:42 +0000 http://scholarlykitchen.sspnet.org/?p=4844#comment-4863 [...] Davis, P. (2009, August 5). Gaming the Rating System. The Scholarly [...]

]]> By: David Crotty http://scholarlykitchen.sspnet.org/2009/08/05/gaming-the-rating-system/#comment-4144 Wed, 05 Aug 2009 14:18:10 +0000 http://scholarlykitchen.sspnet.org/?p=4844#comment-4144 Exactly right. Promotion should be transparent. There’s a reason newspapers and magazines label those fake articles as “advertisements”, and it’s even more imperative for scholarly publishers whose business depends on their reputation.

It’s something that worries me about all the proposed substitutions for the impact factor for judging papers and journals. The impact factor is fatally flawed (the lack of transparency in the “secret formula” makes it impossible to judge), but it’s not easily gamed by running an autovoter or an autodownloader program, or setting up a shill account to give yourself good reviews. Yes, it can be (and is) gamed, but like most things on the internet, the ability to game scales way up for the proposed systems.

]]> By: Philip Davis http://scholarlykitchen.sspnet.org/2009/08/05/gaming-the-rating-system/#comment-4143 Wed, 05 Aug 2009 13:43:57 +0000 http://scholarlykitchen.sspnet.org/?p=4844#comment-4143 Promotion versus Manipulation

Good point, Kent. There is a fine line between promotion and manipulation. We believe that promotion (even self-promotion) is acceptable but manipulation implies some form of deception.

The use of pseudonyms implies a deceptive act, as does the payment (money or in-kind reviews) to the reviewer.

]]> By: Kent Anderson http://scholarlykitchen.sspnet.org/2009/08/05/gaming-the-rating-system/#comment-4142 Wed, 05 Aug 2009 13:31:04 +0000 http://scholarlykitchen.sspnet.org/?p=4844#comment-4142 It’s interesting that the engineering of the Web can help drive and reveal this simultaneously. But it’s nothing new. Authors have been trading book blurbs for decades now — I especially love the “early praise” comments on books, which I think are very often blurb barters. Attention isn’t easy to generate, the clutter is not easy to cut through. But there is a line between the legitimate and the suspect. Unfortunately, it’s like the batter’s box in baseball — each player scuffs it just enough so that by the middle innings, you can’t tell where the lines once were.

]]> By: David Crotty http://scholarlykitchen.sspnet.org/2009/08/05/gaming-the-rating-system/#comment-4140 Wed, 05 Aug 2009 12:38:45 +0000 http://scholarlykitchen.sspnet.org/?p=4844#comment-4140 Back in 2004 Amazon’s Canadian site had a glitch where they revealed all reviewer names and we got an inkling of how that system was being gamed.

]]>