Protest Votes and Video Games: The Metacritic Mambo

Metacritic. The site I love to hate.
Metacritic. The site I love to hate.

I'm a fairly committed gamer: I play games, mods games, write about games, and follow game industry news and if there's one thing I hate more than a crappy checkpoint save system it's protest votes on Metacritic. You know what I mean, those random strings of 0's and 10's that turn the player review section into a dung-heap of subjective butt-hurt and sycophantic butt-sniffing.

Now, I understand that many players are unhappy about their experiences with a game and want to vent. I understand the pain of plopping down $60 on a new game you've been waiting ages for only to discover that the pristine mirror-like reflectiveness of the disc has somehow magically transformed into a steaming turd 15 minutes into the game. But, seriously, if you're going to Metacritic to express your distaste with a big fat 0, you have no one to blame but yourself if shoddy and unprincipled game journalism rules the day. In comparison to fan-boys and haters, even the worst game reviewers come across as paragons of Truth and Justice. If you are one of these people, then you are contributing to the sad, sorry state that is game journalism today. I hope that seems harsh, because it's supposed to be.

See this, haters? Not helpful.
See this, haters? Not helpful.

Warning Signs That You're About to Lose Your S***

Let me know if any of these situations trigger a ring-tone for you:

  • your favorite game franchise tries to appeal to a wider audience and removes all of the interesting features that you liked about the previous games
  • the game you've been waiting for months (or years) for suffers from a number of unfortunate bugs on release that completely ruin your first experience
  • the game is surprisingly short but costs as much as any other game
  • you own a PC and the controls seem better adapted to a console or vice versa
  • the developers think it's a good idea to release day one DLC and put the content on the disc but charge you to unlock it
  • the game has prohibitive DRM that makes you feel like a criminal, guilty until proven innocent
  • popular review sites hand out 100's the way your Aunt Edna hands out last year's Hallowe'en candy but you're still wondering when the game is supposed to get fun

See this, fanboys? Still not helpful.
See this, fanboys? Still not helpful.

Obviously, these are real problems and need to be mentioned in a conscientious review. I'm not here to minimize them, and, taken together, they might constitute giving the game a very poor rating.

But none of these issues on its own is a reason to give a game a 0 out of 10. If you give a game a 0 out of 10 because it does ONE of these things, or even two or three of them, you're just being a douche, plain and simple. If you find yourself on the other side of the fence, defending your favorite games with a zeal that only a honey-crusted nutbar could love, you're contributing just as much to the problem as the haters. Your reviews are just as biased and just as useless to other readers.

Let me know if any of these situations rings a bell:

  • you bought a game and you love it, and you want to share your enthusiasm with others so you go to Metacritic and you see that a bunch of d-bags have given it 0s

This is not a justification for giving the game a rating of 10. It just inspires other people to vote you down with more 0s, creating a vicious cycle of douchyness.

If players can't be trusted to give an objective review, other people will get paid to do it. Those people are called professional game reviewers. And some of those reviewers are corrupt.

Those corrupt reviewers get to keep their jobs because player reviews, on average, are even worse than theirs are so people keep going to their sites. You see how that works? When it comes right down to it, most people would rather take their chances with a sponsored review than read a player review because, on the whole, players are even less objective than the reviewers who depend on ad revenue from the companies whose games they review.

Long series of 0s and 10s do not make the review more balanced, they make the review more worthless and they keep corrupt reviewers in business because, honestly, who thinks that "0: This game sux!" or "10: This game is perfect!" is a good substitute for a page of detailed information about the game even if the final score seems hopelessly biased?

The professional reviewers have to at least mention details about the game to fill up the whitespace on their monitors, and sometimes they mention pertinent details, something you've probably neglected to do in your drive-by voting. All you've done is taken liberties with the review process and demonstrated your complete inability to be any more objective than the reviewers you claim to hate. If you don't have anything good (or objectively bad) to say, please just shut the hell up so that people who do have something to contribute have a chance to speak.

Yes, corrupt reviews are a problem; but rate-rape is not a solution. It behooves you to be the better man/woman.

See this, professional game reviewers? Really not helpful. Get your s*** together.
See this, professional game reviewers? Really not helpful. Get your s*** together.

How People Use Metacritic

To illustrate the effectiveness of your protest voting, let me describe for you how I (and I suspect many other people) use Metacritic:

The first thing I do is ignore any review from a 'professional reviewer' that gives a game a score of 100. (This can be tricky, since many reviewers use different metrics, so it depends a bit on the reviewer, but I'm wary of sites that hand out too many 10/10s or 5/5s, too.) Frankly, I've played a lot of games, but I've never come across a single game deserving a 100. I doubt I've even played any game that I would give a 95 to. I don't think that's harsh, I think that's objective, and if you're a 'professional reviewer' and you're handing out 95-100s for every major game released by a major developer, I'm seriously doubting your objectivity or your credentials. If you only have one review like that a year, maybe I'll give you the benefit of the doubt. Maybe.

The second thing I do is ignore any player review that is a 0 or a 10. If you're giving a game a 0 or a 10, I pretty much will not read your review. The only reason to give a game a 0 is if it doesn't even load up when you take it out of the box, and no one with the same version of the game can get their copy to work either. I have never, ever seen a legitimate reason for a 0. If you're giving a game a score of 0, I know you're a crank voting for political reasons, not an objective reviewer. I don't give in to the demands of terrorists, and I'm not giving in to your opinions either.

If you're giving a game a 10, I'm willing to grant that, in the case that you can't give a game a 9.5 or a 9.7, you might legitimately opt for a 10. I might read your review, but I'm going to be very skeptical if it's less than 500 words strung together in complete sentences expressing coherent thoughts. It's the same criteria I use for judging a 1. I can believe that a game can be so bad that it merits a 1, just as I believe that it can be so good that it merits a 10 (that's a rounding error on scales less than 100), but I'm going to unconsciously give your reviews less weight because I tend to think there's probably some bias involved. The number of games that, in my opinion, deserve a 1 or a 10 is very, very small so you're going to have a hard time selling me on your opinion. If you're going to go for a score like this, you'd better have some good ammunition because 'I didn't like it' and 'best game I've played all year' are not going to cut it. If you don't have a single, specific reason for giving the game a 1 or a 10, I'm ignoring your vote.

When you get into 2s and 9s and everything in between, I'm open to your opinion. Personally, I think there are a lot of 7s out there that should be 5s so I tend to think that 70 has become the new 50. That's unfortunate because it reduces the usefulness of a 100 point rating system (by making it more like an 80 point rating system). In my mind, a game that scores, say, a 40, should be worth playing if you're a big fan of the game's genre. About half of the games that get made should fall below 50, and about half of them above. As it is, that cut-off has shifted to about 70, which means that there is less room to make fine distinctions among the better games.

How To Write Useful Game Reviews

The point of this rambling rant (and I know you don't really care what I think of what you do on Metacritic but you should because there are plenty of people out there who think like me) is that when you rate a game for political reasons instead of objective reasons you aren't doing anyone any good and you just come off sounding like an ass. You might think that you're hurting game sales by voting a 0, but the fact is -- and everyone knows this, except, apparently for the 0 voters themselves -- a 0 vote is a crank vote, it's a butt-hurt vote and nobody gives a s***, certainly not the publishers and developers, certainly not the professional reviewers, and certainly not the vast majority of the people who use Metacritic. No one takes your vote seriously. You're a joke.

If you really want to hurt a company that you feel hasn't treated you fairly, write a well-reasoned review and give the game an objective rating. I will pay attention to you if you can support your arguments with evidence from the game and your rating seems fair based on the points that you make. You might even convince me and other people not to buy the game based on reasonable arguments. And guess what? Some of the more open-minded developers might care, too, which is what you want, isn't it? What I can guarantee you is that nobody is taking your 0 seriously. You are literally just throwing away your vote.

On the other hand, if you're giving a game a 10 and you provide no objective criteria for that vote, you will sound like a slavish drone and you deserve to be reviled. You deserve to have people down-voting your favorite games with a relentless tattoo of blood red bullet-hole zeroes.

If you really care about a game, and you think that it's being given the short end of the stick, rate it fairly and back up your claims with evidence from the game. I've read more 7/10 and 6/10 game reviews that won me over than I have 9/10 or 10/10 reviews. You know why? Because those reviewers took the time to write a good review that gave me enough information to have reasonable expectations about a game. I knew what I was getting into, and I could appreciate the game for what it was and it made me want that experience, even if it wasn't perfect. It's not fan-boys that sell games, it's fair reviews.

Roll Your Own Bias-Correction Filter for Metacritic

Because Metacritic doesn't have an easy way to filter out protest votes or publisher influence on professional reviewers, and I can be incredibly OCD at times, I've developed my own formula for correcting for critic and player bias in review scores. Although it's based on my interpretation of hundreds of reviews over dozens of games, this formula is highly subjective; it's not intended to be even remotely scientific and can't be considered to be based on extensive data analysis; it's merely intended to be a useful heuristic.

Here's my formula for adapting a Metacritic score to a score that doesn't make me want to throw up in my mouth:

  1. First, multiply the player review score by 10 so that both scores are based on 100. Let's take SSX for the Xbox 360 which has a user score of 6.6 as of this date. 6.6 becomes 66.
  2. Now, subtract the lower score (usually the user score, in this case 66) from the higher score to get the difference: the review sites gave the game a 82, so the formula is 82 - 66 = 16.
  3. Next, treat the difference as a percentage and multiply it by the difference: in this case, 16% of 16 = 0.16 * 16 = 2.56. Round that to the nearest whole number: 2.56 = 3.
  4. Now, add this number to the lower of the two original scores, in this case, to 66: 66 + 3 = 69.
  5. Next, get the total of the new score and the higher of the original scores: 69 + 82 = 151.
  6. Now divide that by 2 and round to the nearest whole number to get the adjusted average: 151 / 2 = 75.5 rounded to 76. My adjusted Metacritic score for SSX is 76%.

It might seem pointlessly pedantic to do the calculations with the percentages in the middle: if you average the critic reviews with the user reviews you end up with 74%, which isn't much different from 76%. For games with small differences between critical and user reviews, there won't be a lot of difference. Where this formula is useful, however, is with games that have large differences between player and critic review scores, like Call of Duty: Modern Warfare 3 and Mass Effect 3.

You might also be wondering why I'm introducing a bias in favor of the higher scores by taking a percentage of the difference instead of just averaging the results. That's because giving a game a score of 0 is usually more unfair than giving it a 10: if the game 'deserves' to be a 6, 7, or 8 out of 10 and you give it a 10, you're only corrupting the data by 2-4 points. If you give the same game a 0, you're corrupting it by 6-8 points. This modifier helps to account for this discrepancy without ignoring completely the impact of a large number of negative reviews. (While it's possible for a very poor game to be highly overrated, eg. a 2 or 3 being given a 10 rating by a fan, it doesn't tend to happen very often so it's safe to ignore.)

Here are a few more examples to give you a better idea of the results (all examples are for Xbox 360 for simplicity):

  1. Deus Ex: Human Revolution: Critics: 89, User: 82, difference is 7 points. 7% of 7 is 0.49. Round to 0. Add 0 to 82 to get 82. Add 82 to 89 and divide by 2 = 171 / 2 = 85.5 rounded to 86. On games with small differences between player and professional reviews (7 or fewer points of difference), the result is an average of the two scores. Sometimes players agree with critics and the review process doesn't seem broken at all. (Portal is another good example.)
  2. Skyrim: Critics: 96, User 84, difference is 12. 12% of 12 is 1.44 rounded to 1. Add 1 to 84 for 85. Add 85 to 96 for 181. Divide by 2 for 90.5 round to 91. Skyrim's a great game, but it's clearly not perfect. I think a 91 is much more accurate than the 100 that many reviewers gave it. (I'd rate it around an 85. Much as I love the game, there are way too many little things and big things that could have and should have been fixed.)
  3. Mass Effect 3: Critics: 93, User 50, difference is 43. 43% of 43 is 18.49 rounded to 18. Add 18 to 50 for 68. Add 68 to 93 for 161. Divide by 2 for 80.5, round to 81. I can believe that Mass Effect 3 warrants an 81 based on the reviews that I've read. It would have scored much higher if the ending hadn't been such a huge disappointment technically and artistically.
  4. Call of Duty: Modern Warfare 3: Critics: 88, User: 32, difference is 56 points. 56% of 56 is 31.36. Round to 31. Add 31 to the lower score, 32, for a new player score of 63. Add 63 to 88 for 151. Divide by 2 = 75.5, rounded to 76. I can believe that the game warrants a 76. A solid, formulaic title that doesn't innovate. (The same score I gave Titan Quest.)
  5. Dragon Age 2: Critics: 79, User: 43, difference is 36. 36% of 36 is 12.96 rounded to 13. Add 13 to 43 for 56. Add 56 to 79 for 135. Divide by 2 for 67.5 rounded to 68. That score is fairly close to the score JohnGreasyGamer gave Dragon Age 2.

This formula works pretty well for me at reducing bias in Metacritic ratings. It's not perfect, obviously, but I find that the results that I get are a lot closer to my own ratings, which, obviously, I trust (though they are obviously equally biased from anyone else's perspective). You may want to tweak the formula a bit to bring it in line with your own views, or -- more likely -- you think the whole thing is bogus and a lost cause.

Personally, I use these calculations to help me understand game trends with minimal bias from either paid journalists or rage-voting, which can be useful if I need to get a quick 'feel' for a game but don't have time to play it (which happens more often than not). I put it out there for anyone else who might be looking for a quick and dirty solution.

User Reviews on Metacritic

How often do user reviews on Metacritic affect your decision to buy a game?

  • All the time. I think user reviews are valuable.
  • Sometimes. I check them out, but don't take them very seriously.
  • Never. User reviews are completely worthless.
See results without voting

More by this Author

Comments 2 comments

j-u-i-c-e profile image

j-u-i-c-e 4 years ago from Waterloo, On Author

I think the best reviews are about 50% objectivity and 50% subjectivity. I don't think it's possible to be any more objective than that, because so much depends on taste. But there are way too many reviews out there that either completely ignore problems or completely ignore redeeming virtues.

In a sense, of course, all of the numbers come out of 'thin air' but some of those numbers still seem a lot more accurate than others. If it was entirely random, eventually, all of the games would end up with the same rating, and that's clearly not the case. There must be some kind of basis that people use when picking these numbers because many people do agree fairly closely on a lot of games.

As to whether or not review scores should have numbers attached to them at all, I think that they are important because people want to know if, after all is said and done, the reviewer actually enjoyed playing the game. They sum up the reviewer's overall impression in a way that his article can't. He might write down all the things he liked, and all the things he didn't, and that's important for you as a consumer, but it's the final score he assigns that tells you how well he thinks the mix worked overall. It's a way of quantifying the ineffable, like how satisfied they were with the final result. Sometimes that overall feeling gets lost in a review because so much of the time is spent explaining specific features and problems. I think that's the reason why ratings were invented in the first place.

Scores alone, of course, should never be a basis for making your decision, but should be treated like any other element of the review. People definitely need to lose their fixation on numbers and spend more time reading. I'm a little different from most people though because I use review scores for more than just making purchasing decisions. I also use them to follow industry trends, spot controversies, note disconnects between consumers and producers, etc.

Thanks for replying. I appreciate the feedback.

William157 profile image

William157 4 years ago from Southern California

Wow. This article was written with white-knuckled passion. I'm impressed that you created an entire equation for finding the "real" score of games, but I'm worried that it's still limited by the fact that most people are pulling scores out of thin air. In fact, some might argue that scoring games on any scale is inappropriate. Hey, that might be a good topic for a hub!

Voted up and interesting!

    Sign in or sign up and post using a HubPages Network account.

    0 of 8192 characters used
    Post Comment

    No HTML is allowed in comments, but URLs will be hyperlinked. Comments are not for promoting your articles or other sites.

    Click to Rate This Article