Advertising, Metrics and Analytics, Sociology

Gaming the Rating System

App Store
Image via Wikipedia

Recently, I received the following email from a friend:

Dear iphone/ipod wielding friends,

I need a 99 cent favor. I just published my new app and I need a little help getting noticed. Unfortunately, the app store has a popularity driven system. To do well, you need to get into the top 100 list. Would you guys buy the app? I will generously reimburse you! If you can do it today, that would be great! You can use this link …

While I removed some information from the email so as to protect the identity of the sender, I have to tell you that my friend is a smart man and well-versed in online economic systems.  Perhaps this is why he sent out the email.

The gaming of rating services is not new.  An analysis of Amazon.com revealed that hundreds of product reviews are merely copied in part (or in whole) from other product reviews.  Sellers and buyers using eBay’s reputation system are constantly under scrutiny against manipulating their own ratings.

Nor are academics immune from ratings manipulation.  University administrators often engage in behavior designed to increase their institution’s ranking in US News & World Report, and professors are known boost their own ratings on the online website, RateMyProfessors.com, by posing as students.

More recently, Elsevier was admonished for offering $25 Amazon gift cards to academic contributors, their colleagues, and their students willing to give  a 5-star rating to a newly published textbook.

My point here is that recommender systems are game-able, especially when contributors are anonymous or when the digital marketplace allows the easy creation of pseudonyms and sock puppets.

The effects of early promotion  are clear to those in business as well as academia:  downloads encourage further downloads, readership begets more readership, and citations beget more citations.  In social environments where users see traces of peer behavior, small early advantages can grow to become large successes, which is why glowing reviews, stellar ratings, and top listings are so important early on.

In writing this piece, I contacted my friend to reflect on his own email request.  Would he be brazen, bashful, remorseful?  He responded:

I see my email as a piece of shameless self-promotion with friends, like sending an article to colleagues, or telling someone you just published a book (and of course implying that buying it would be a good idea) … trying to get friends to buy the app is a common practice, but scarcely new to the online marketplace. Nor is getting your product in the best seller list new. Publishers of books and music have been trying to do it for decades.

Postscript: If you like this blog post, forward it to as many friends as possible!

Reblog this post [with Zemanta]

About Phil Davis

I am an independent researcher and publishing consultant specializing in the statistical analysis of readership and citation data. I am a former postdoctoral researcher in science communication and former science librarian. http://phil-davis.org/

Discussion

5 thoughts on “Gaming the Rating System

  1. Back in 2004 Amazon’s Canadian site had a glitch where they revealed all reviewer names and we got an inkling of how that system was being gamed.

    Posted by David Crotty | Aug 5, 2009, 8:38 am
  2. It’s interesting that the engineering of the Web can help drive and reveal this simultaneously. But it’s nothing new. Authors have been trading book blurbs for decades now — I especially love the “early praise” comments on books, which I think are very often blurb barters. Attention isn’t easy to generate, the clutter is not easy to cut through. But there is a line between the legitimate and the suspect. Unfortunately, it’s like the batter’s box in baseball — each player scuffs it just enough so that by the middle innings, you can’t tell where the lines once were.

    Posted by Kent Anderson | Aug 5, 2009, 9:31 am
    • Promotion versus Manipulation

      Good point, Kent. There is a fine line between promotion and manipulation. We believe that promotion (even self-promotion) is acceptable but manipulation implies some form of deception.

      The use of pseudonyms implies a deceptive act, as does the payment (money or in-kind reviews) to the reviewer.

      Posted by Philip Davis | Aug 5, 2009, 9:43 am
      • Exactly right. Promotion should be transparent. There’s a reason newspapers and magazines label those fake articles as “advertisements”, and it’s even more imperative for scholarly publishers whose business depends on their reputation.

        It’s something that worries me about all the proposed substitutions for the impact factor for judging papers and journals. The impact factor is fatally flawed (the lack of transparency in the “secret formula” makes it impossible to judge), but it’s not easily gamed by running an autovoter or an autodownloader program, or setting up a shill account to give yourself good reviews. Yes, it can be (and is) gamed, but like most things on the internet, the ability to game scales way up for the proposed systems.

        Posted by David Crotty | Aug 5, 2009, 10:18 am

Trackbacks/Pingbacks

  1. Pingback: An Inflection Point for American Public Libraries | In the Library with the Lead Pipe - Sep 30, 2009

Side Dishes by Stewart Wills

Find Posts by Category

Find Posts by Date

August 2009
S M T W T F S
« Jul   Sep »
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

The Scholarly Kitchen on Twitter

SSP_LOGO
The mission of the Society for Scholarly Publishing (SSP) is "[t]o advance scholarly publishing and communication, and the professional development of its members through education, collaboration, and networking." SSP established The Scholarly Kitchen blog in February 2008 to keep SSP members and interested parties aware of new developments in publishing.
......................................
The Scholarly Kitchen is a moderated and independent blog. Opinions on The Scholarly Kitchen are those of the authors. They are not necessarily those held by the Society for Scholarly Publishing nor by their respective employers.
Follow

Get every new post delivered to your Inbox.

Join 14,399 other followers

%d bloggers like this: