The problem with reviews

I like to think I'm a savvy consumer. Being a student, I don't have a lot of disposable income, which I suppose makes me a little more cautious about what I buy. To this end, I've adopted a general rule that I won't buy a game over about $15 until I'm reasonably sure that I'll be getting my money's worth.

I normally like to play a game for a little while, to get a clear feel for how it plays, before I make a decision. I love that many games on Steam have a free demo that I can try out, but for some reason a lot of games still don't bother to release a demo. I'm sure it's a lot of work to produce a game demo, and I think the bean-counters at the large publishing companies have probably evaluated that it's not worth the investment and they'd rather not bother with hesitant buyers like me.

One might think I could rely on game reviews to help me decide - Magazines and websites publish textual and video reviews. Metacritic uses some mathematical wizardry, compiling multiple reviews to produce 'meta-scores', giving a general overview of a game's review scores. Yahtzee's Zero Punctuation videos home in on flaws that might otherwise go unnoticed by the generally-positive review websites. I don't consider game reviews to be a reliable, accurate or otherwise useful measure of a game's quality, and I hope this post can list the issues I see in them along with some ideas on how to improve things.

A review serves two main purposes to consumers - to describe the gameplay of a title, and critically discuss its quality. For the first, it's pretty easy to explain the gameplay of most games, since they tend to fall neatly into some genre - Anyone familiar with a first-person shooter knows generally what to expect when the top of the review says that Half-Life is an FPS game. The remainder of the task, then, is to give a more detailed description of how the game 'plays'. In my opinion, not even a lengthy video of raw gameplay footage will completely demonstrate the finer details of a game's play experience - fluidity of movement can't really be conveyed unless you've got a controller in your hand, and videos don't often include players running into invisible wars or immersion-breaking bugs.

After explaining how a game plays, the other job of a review is to discuss its quality. I see two critical problems with this task - A conflict of interest between reviewers and their advertisers, and a shallow review scoring system made more shallow by a tendency to score games highly.

A game reviewer is generally part of a business, and most of these businesses are supported in part (often entirely) by advertising. These ads are of course for games, because it's such a perfectly-targeted campaign - Anyone reading a gaming website is probably interested in games, so ads for games are going to be there. This means that games are often reviewed in publications that are deriving income from advertising that particular game. The management of such a publication would surely be under pressure from their advertisers to not publish negative reviews of an advertiser's game, and while I'm sure that most reviewers have journalistic integrity to spare after dismissing such pressure, the conflict still exists.

It was confirmed earlier this year that Jeff Gerstmann was fired from GameSpot after he gave a 6/10 score to Kane & Lynch: Dead Men - the company caving to pressure from Kane & Lynch's publisher Eidos. At the time the review was posted, the GameSpot site had been largely re-skinned with ads for the game. I won't touch on other examples, but this video has a lot of them.

The other serious issue I have with game reviews is their tendency to assign games some numerical score based on the reviewer's subjective opinion of it. It's an issue with readers taking the numbers to be a quick reference of how good a game is, but the problem becomes more serious when the games industry takes these numbers seriously, like in the case of Fallout: New Vegas. Metacritic's popularity is highly concerning since it focuses primarily on these numerical scores, which can vary wildly based on a publication's scoring system and the specific play experience of each reviewer. One Word Reviews parodies this idea by allowing users to propose a single word summarising a game, though its current state has been heavily skewed by the influence of videogame community memes.

Top three one-word reviews for Team Fortress 2.

The increasing popularisation of indie games and diversity of pricing systems has also confused many review systems. A review exists to help inform a potential buyer of whether they will get their money's worth out of a game, so are review scores altered based on the game's price? I think this is an important factor to consider. I was really disappointed in Borderlands because I paid $50 for it on release, and felt like I'd paid too much. If I'd waited for a Steam sale and picked it up at $15, I probably wouldn't have felt so burned by the lackluster experience.

Another problem is that big review sites tend to lean much more heavily to the top of the scale when assigning final scores. One might think that a 50% score average would make sense, but looking at the 50 most recent reviews for Xbox 360 games on IGN gives an average score of 78%. Two games scored below 50%, both 30%. 9 games scored 90% or over, and 24 scored 80% or over.

David Jaffe has strong opinions on this matter.

Resident Evil 6's Metacritic page shows critic review scores ranging from 95% to 20%. Variation like this in review scoring (outside of user reviews) is not particularly common but it shows that these scores can be essentially meaningless.

Review scores are generally attached to the end of a descriptive article explaining the reviewer's experience, and these pieces are often far more reliable and useful sources of consumer information. Unfortunately the simple existence of these numerical scores will mean that many readers may skip to the end for the numerical scores that rank graphics, presentation, gameplay, story, etc. Who came up with these categories, and how are they measured?

While it's difficult to resolve the issue of conflict of interest in mainstream reviews, at least the issue with review scoring is addressable by considering new methods of rating games.

There is certainly an audience for concise game scoring and I feel that, if done right, there's also a reasonable chance of a system being useful. First let's talk about scoring subcategories, which help to add some granularity to what can otherwise be a very obtuse scoring system.

Jesse Schell's excellent book The Art of Game Design divides games into four basic elements. I've lost my copy somewhere, so I hope I've got them right:

• Mechanics - the rules and systems of the game world,
• Story - the plot, characters and writing of the game,
• Aesthetics - how the game looks and sounds, and
• Technology - the underlying systems supporting a game.

These elements are well-defined categories that can cover all aspects of a game experience, and once a reader understands them correctly should be able to identify the importance they place on individual elements. They can be further subdivided, and upon consideration I might add a fifth category - World - to cover level design and world design as a whole.

On the other side of the coin, the player is also a key element in determining whether that player will enjoy a game. It seems as though mainstream game scoring doesn't really make an effort to address this, though many review articles will touch on this important problem. Players come from different backgrounds with their own skill levels, likes and dislikes. This is another area that Schell touches on in his book, but this page gives a brief outline of some player types.

Finally, game reviews may have been able to gloss over pricing systems in the past, but with the diverse and ever-changing landscape of game monetization systems of today, reviews certainly now need to address some kind of 'bang for your buck' score. Other important issues to consider are whether a game's payment system interferes with game flow or disrupts competitive balance (the 'pay to win' problem).

I intend to review games on this blog: classics from the past, games with cool ideas and maybe even some mainstream ones too. With the above issues in mind I'll do my best to provide an impartial and rigorous assessment of a game's quality, and hope I can come up with a better system for assigning scores to a game. It's clear to me that these scores are being taken far too seriously for how subjective they are, and I hope I can do my part to affect some change in the way games are reviewed by mainstream publications.