Social literacy – does ‘karma gaming’ matter?

I have a question for all of you smart people out there. Does the fact that some people ‘game’ karma or rankings or anything you can count in social networks matter?

Here’s the context – I’m doing some work with the Drupal crew and at first blush it seems (perhaps obviously?) that Drupal.org is actually much more of a ‘social’ site than a ‘content’ site, and that many of the social and content issues we need to address might be helped along by making the activity that community members are undertaking visible – which essentially becomes a kind of ‘karma’ (a la Slashdot)

Karma is the sum of your activity on Slashdot. This means posting, moderation, story submissions. It’s just an integer in a database. The tiers are Terrible, Bad, Neutral, Positive, Good, and Excellent.

The obvious objection to this is gaming – that people will behave in a way that will increase their karma which is not necessarily in the best interest of the community at large.

There are a number of ways that you can design karma systems so that they are less likely to be gamed (see again Slashdot):

People like to treat their Slashdot Karma like some sort of video game, with a numeric integer representing their score in the game. People who do this simply are missing the point. The text label is one way we’ve decided to emphasize the point that karma doesn’t matter.

and

Yes. Karma is now capped at “Excellent” This was done to keep people from running up insane karma scores, and then being immune from moderation. Despite some theories to the contrary, the karma cap applies to every account.

And, of course, the well documented removal of the ‘top diggers’ page from Digg is another example of removing the incentive to game karma.

What I wonder about, though, is this:
  • is it just a minority of people who are compelled to have enormous numbers next to their names? 
  • is this minority behaviour *really* disruptive to everyone’s experience of the system?
When I think about my experience of Twitter – I know there are a bunch of people out there who are very interested in having huge lists of followers. I guess it’s an ego thing. For the rest of us though, we get our value from Twitter from a much smaller group of more carefully selected individuals. 
And we’re literate about ‘the numbers’ when evaluating people to include in our network. A common behaviour is to compare the ‘following’ to ‘followers’ numbers – if the ratio is screwed far toward ‘following’ then proceed with caution (or, more likely, don’t follow!)
My gut feeling is that:
  • yes, *some* people will game the system. No system you design will ever stop this.
  • making this gaming visible and traceable (you can see what people have done to achieve their ‘number’ or ranking) means that people can be discerning about the value they place on ‘rankings’
  • if people have high rankings, chance are they’ve been busy/loud in the community – their true value is really on the value of the noise they’ve been making (hopefully good noise)
  • showing rankings in order (a la Top Diggers) should be avoided at all cost.
What do you think?

13 Responses to “Social literacy – does ‘karma gaming’ matter?”

  1. Tom Armitage September 3, 2008 at 4:15 pm #

    I think you’re roughly right, in that:

    - yes, some people will always game the system.

    - making the reasoning behind the score available helps increase transparency, but many users won’t delve that far – you put a number by something, and people want to make it bigger.

    - some people with high scores will be valuable, contributing members of the community. The harder it is to game karma, the more likely that will be.

    - “high-score charts” aren’t necessarily always a bad thing… but they’re like a red rag to a bull when it comes to gaming karma.

    I think the most useful “karma”-type metrics tend to come from compound calculations – not just a quantity of posts or comments, but quantity-of-comments-per-post (things people are interested in) or diversity-of-audience-of-posts (you’re posting content that a wide variety of people find interesting, rather than generating tiny conversations). Compound scores – eg, Flickr’s interestingness – are much harder to game because they’re hard to work out, which is why I think your second bullet-point might not necessarily be true. You stop trying to perform the actions to get a good score, and instead use the site well – and discover your score goes out.

    Is this minority behaviour damaging? It depends precisely what behaviour they’re taking, and how much it deviates from the desired behaviour of users on the site. If, by karma-gaming, they’re still contributing positively (ie: engaging in lots of conversation, being helpful), that’s not a problem. If the karma-gaming takes the form of, I don’t know, something like mechanized, major, spamming attacks… then clearly it is damaging. The fact it’s a minority of users who do this doesn’t mean it can be majorly disruptive.

    Also: consider how much influence karma-gaming has on new and existing users of the site. Regardless of whether or not you display a “score” of some sort, if the community decides that karma is valuable and relevant, karma-gaming (as long as it’s not detected) may be condoned as a valid-strategy for using the site. By contrast, if the community has declared the karma-system irrelevant or bankrupt, that will be conveyed to new users, and so karma-gaming has no value because karma itself is worthless.

    Um, that’s all I’ve got, so far…

  2. LKM September 4, 2008 at 6:57 am #

    Do people who game their karma harm? I think it depends on the specific site. If getting high karma is part of the goal of using the site, then people who have incredibly high karma actually harm others, because they discourage them by setting unattainable goals (think playing an Xbox live videogame, doing incredibly well, and ending up ranked at number 1,964,453,345 or something – it’s discouraging).

    If getting high karma means you decrease the value of the content for others, it’s obviously harmful, too. For example, sites like digg rely on having interesting links on their frontpage. If somebody were to cheat by having tons of sock puppet accounts which upvote his own comments and links, that would decrease the quality of the upvoted content on digg. Another example: If gaming Twitter means that some people add everyone they see to their friends list, in the hopes that he or she will reciprocate, that results in a ton of spam for everyone from all the gamers.

    If gaming the system does not harm others, however, it doesn’t really matter, I guess.

    The best solution would probably be to try to make sure that users who game the system actually improve the system; i.e. the only way to game the system is by contributing positively to the site.

  3. Boris Mann September 4, 2008 at 7:58 am #

    A lot of it comes down to eyeballs and links. There are some people that will invariably “top post” a comment on every single front page posting. It’s technically on topic, but it’s so consistent, and the affiliate link in the signature so obvious, that it’s highly annoying…

    I think visible indicators of *community* rankings would be way more useful. For instance, I often use the contact form to email people “offline” to say thanks for a forum posting or if I’ve noticed them being particular active.

    I love the “cheers” system that 43 Things / Places / etc. uses.

    You could cheers users or projects — I think those are the two main data objects that would benefit from this.

    So, perhaps a user initiated “thanks” of some kind. Making it unique to Drupal, with visible badges (although not, necessarily, absolute rankings visibly displayed — more like levels).

    OK, and while we’re on badges, perhaps certain people can award different kinds of badges, or for special events.

    In summary … sheer volume isn’t a good indicator, let’s let the community reward each other.

  4. Gordon September 4, 2008 at 8:08 am #

    The Stack Overflow community (still in closed BETA I think) handles this by allocating different weightings to activities.

    So whilst I could rack up a high karma score there, it would require so many repetitive and tiny transactions with the website that I’d be discouraged from doing so… the medium ‘scored’ transactions require some buy-in and some work, and the top level transactions are, essentially, awards given to you by others.

    Yes it can still be gamed, everything can, but as you say, the transparency here is key. If I’m reading something by someone with a high karma/score then I want a quick indication of how (if!) they earned it!

    Taking the above system into play you’d soon see if a high scorer had no top and medium scores, or vice versa, and adjust your thinking accordingly.

    P.S. Will see you later this month in Edinburgh!

  5. Kars September 4, 2008 at 9:08 am #

    This is a very interesting dilemma with which I have been struggling in a few projects too.

    Some very valuable comments have been made already. I thought I’d contribute some stuff less focused on the formal aspects of social software and more on the experiential (play) level. Some things to consider:

    * any choice you make about the formal aspects of the system will influence the perception people have of the community’s social contract
    * besides the official rules of the system, people socially construct actual rules
    * what you call “gaming” is an individual or group of players proposing their own set of actual rules

    If you haven’t already, I would recommend you pick up ‘Rules of Play’ and read the chapter ‘Games as Social Play’. Some very good stuff on these issues.

  6. Lloyd Davis September 4, 2008 at 9:41 am #

    Leisa, I don’t have an answer, but my initial reaction to what you wrote is surely there are ways of “making the activity that community members are undertaking visible” other than reducing the complexity of their activities to a single integer.

    If you have to go for something this simplistic, I would think more along the lines of blip.fm and their props (each user starts with a number of props they can give away to other DJs and when they receive them, they can pass them on) it’s still susceptible to gaming but of a more complex sort and one that requires co-operation with others.

    Which reminds me that what we’re talking about here is “undesirable” gaming. You don’t want to stamp gaming out – some element of playing the game is what makes social sites… err… fun!

  7. Robert Castelo September 4, 2008 at 11:10 am #

    What I would love to see is a karma system where the top 50 most respected community members are given 100 points each, they can then reward other members with points, who can then do the same to other members – so that karma trickles down based on people appreciating each other’s contributions rather than an arbitrary algorithm.

    Note: The amount of points a member has never goes down, and is used as a limit of how many points they can reward.

    In effect it becomes a karma currency.

    Is there any sites with a system like this?

  8. James Breeze September 4, 2008 at 11:32 am #

    Nice post Leisa! The gaming aspect can be quite addictive!

    I’ve been a little bit strategic about what I do in professional social networks. Some of those strategies mean that I do keep track of what is going on. Kinda like KPIs.

    I actually would like a tool to track my ongoing games in one spot. I’d be nice to see them all on one page!

    Anyone know one or got any ideas?

  9. gaele September 4, 2008 at 3:27 pm #

    Read #5:

    “any choice you make about the formal aspects of the system will influence the perception people have of the community’s social contract”

    and the “cheers” system Boris mentions in #3 sounds like the way to go. Stress the “giving” aspect of karma, not the “gaining” aspect.

    Also see ubuntuforums.org, e.g. http://ubuntuforums.org/showthread.php?t=840040 (“The Following 6 Users Say Thank You” to this user).

  10. Boris Mann September 4, 2008 at 3:54 pm #

    @Robert: I wouldn’t want to see a “star system” like that. The whole point is that you can come in anywhere into the community, start contributing, and get recognized.

    @Kars “any choice you make about the formal aspects of the system will influence the perception people have of the community’s social contract” — great quote, and that’s exactly why I mentioned the cheers system — it reinforces the social norms already in the community (E.g. thanking people by mentioning their name in commit messages).

    And, of course, I think cheers systems are quite different than points systems, so the whole gaming aspect may not apply at all…

  11. leisa.reichelt September 4, 2008 at 8:04 pm #

    thanks so much everyone for your comments – it makes for a really interesting read.

    I’ve been really taken by the simple ‘props’ system that Blip.fm uses (which sounds similar to the ‘cheers’ or ‘thanks’ systems you’ve described. What I love about Blip.fm is that the more ‘props’ you receive, the more you have to give. There is something very positive about that, which I really like.

  12. James September 5, 2008 at 3:57 am #

    Hi Leisa, great post and great discussions sparked. We’re just working out how to use the existing Drupal modules to flesh out our ideas about a reputation currency. Something deeper than e-Bay star ratings and more reflective of the ways we actually allocate and manage reputation in relationships.

    I’d love to talk to you about how we can make this custom development work for the overall Drupal community, if it’s of value. Please send me an email and we’ll make that happen.

  13. David Geilhufe September 6, 2008 at 5:56 pm #

    Begs the question of what earns you karma. In communities where karma doesn’t take a lot of work of high value to learn, gaming seems to be more prevelent.

    If we set up karma in D.org to mean code contributions, handbook contributions, etc., then it takes concrete hours of work to earn it and therefore lowers the probability of gaming.

    One of the key elements of Drupal is that new people can become key people by doing work. Any karma system must reflect and identify those up and comers / recently arrived leaders.