Video Game Reviews are Broken @ Kotaku

magerette

Hedgewitch
Joined
October 18, 2006
Messages
7,834
Kotaku has an editorial up on the subject of game reviews and why they don't seem to work properly, their theory laying the blame on the common concept of the numerical score:
...But after reading countless video game reviews over the years,...it's become more and more obvious that the video game review system is dated, limiting and even, at times, unintentionally condescending....
...If there is no such thing as a perfect game, then why the hell are you scoring out of 100? It's not just PC Gamer that thinks this way—most publications, even those who do give out "perfect" scores, do so begrudgingly. It's as if the developer has somehow cheated and broken their system.
The movie reviewers solved this problem a long time ago. That's why most adopted a simpler rating system in which a 4-star movie didn't imply "perfection" but supreme excellence. In most cases, games are penalized through being divided by a sum that they can never possibly reach. What does that make a 94 or a 9.5 then...is that our mortal interpretation of perfection? Is that the closest we can fly to the sun before our wings melt and we're doomed to playing Spongebob Squarepants XVI for eternity?...
...The fundamental problem with game reviews is that they're analyzing products, not pieces of art. Or more clearly stated, art reviews decide if something is worth your time; game reviews decide if something is worth your money. Let's go back to our convenient film comparison for a moment. When do reviewers ever complain that a movie is only an hour and a half long? They don't, because length as a value proposition is generally never affiliated with art...
Games are viewed as consumable goods meant to entertain for X hours at X amount per X dollars. There's an interest in durability (replay value, multiplayer), functionality (controls, camera), interface (HUD, menus), sex appeal (graphics) and accessibility (difficulty level)...
We really do direly need the video game review industry because there is just too much volume for gamers to get prioritize[d] alone, even with word of mouth, message boards, etc. We need people and organizations that can plow through multitudes of games to pick out the gems and crap on the...crap.
But we need them to analyze the subject in a critical light, judging a game's intrinsic value over its dollar value.
More information.
 
Joined
Oct 18, 2006
Messages
7,834
Bah, this could easily be fixed if reviewers stopped using decimal values or ratings out of 100. All the decimal values like 9.6/10 do is make it hard on a reviewer when a clearly better game comes out.
"Uh-oh, I rated Shooter 2 a 9.9/10, but Shooter 3 is clearly better! I'll have to rate it 9.95/10 - that should do it."
None of this really matters, since all readers really look for is where it generally falls: is it a 9-10, 8, 7, 6, or below? Reviewers should rate using simple ranges like Crap, Average, Above-Average, Good, Great, and Excellent instead of breaking it down to such tiny, nitpicky fractions.
 
Joined
Sep 27, 2007
Messages
71
Location
Virginia, USA
How do you put down impressions into something *anybody* can understand ?

We must decide upon a unified language first.

Numbers *can* be such a language, or at least most people seem to believe that.
 
Joined
Nov 5, 2006
Messages
21,908
Location
Old Europe
It occurred to me when I got a paper back from class and I was looking at votes at NWVault that numbered games reviews more closely resemble test scores and class grades than a "5 being average" scale. Less than 50% means failure here in NA.
 
Joined
Oct 19, 2006
Messages
5,212
Location
The Uncanny Valley
Exactly. Everyone (in general) sees it the same way, otherwise scores at Gamerankings or Metacritic from gamers would cluster around 5/10 -- but they don't. In the main, they're remarkably close to the critics' scores.
 
Joined
Aug 30, 2006
Messages
11,842
Location
Sydney, Australia
Just to add my two cents, I always thought that the best system for grading a game was the system PC Games used. (This was way back in the day where there were five or six magazines covering the PC scene. Later on PC Games would be rolled into PC Accelerator, which a year or two later was rolled out of existence, but that's another story...) It was simple, easy, and understandable- A-F, with pluses and minuses for precision. Just like in grade school. A was great, B was good, C was not great but playable if you really liked the concept/genre/developer/etc., D was bad, F was simply unacceptable. No E because you could add a single stroke to an F and... uhh... get spanked LESS severely by Dad, I guess.

But the thing is, you KNEW how it worked. Sure you'd get into arguments on how game X was scored too high or too low, but everyone understood what the score MEANT- none of this "Shouldn't 5.0/5.0 mean 'gaming nirvana'?" or "Is average 5.0 or 7.0?" crap. The first rule of writing is to talk in a language your audience understands. But no, we have to quantify instead of qualify because it's "more accurate", so we wind up confusing everyone with our accuracy.

If I ever make a game review site, I'll be using a 10.00 system and just generate scores with a random number generator. Play it up, too- trumpet the advantages of PUDSS, the Phenominally Useless Decimal Scoring System, which with much less effort produces scores equal to or better then any other gaming site on the web with accuracy. Go into a long-winded discussion about how it incorporates all aspects of game design into a byzantine scientific formula modified by genre, system, and relative hype index (RHI), then "rolls a hundred-sided die and inserts a decimal point to determine the final score." Add a site feature that allows you to rank all reviewed games based on score, then point them to the page expounding on PUDSS when they complain that Classic Game Y got 3.84 while Crapware Game Z got a record 9.76 for the genre.

I mean seriously, guys, does anyone actually read the damn review anymore?
 
Actually, YES!! I do. Check out our reviews here, while we use a numerical score, each number has a corresponding comment.
 
Joined
Aug 31, 2006
Messages
12,806
Location
Australia
Reviews ought to provide readers with information they can use to make a decision. But some readers want more than that. They prefer to be told what decision to make. In that respect, numerical scores are really not much more than ornate "thumbs up / thumbs down" decisions.

The sad truth is there are quite a few people who struggle with decision-making throughout their lives and rely on help every step along the way.
 
Joined
Nov 11, 2006
Messages
1,807
Location
Orange County, California
The scoring system is really not among my major beef with game reviews. It is a really minor problem compared to reviewers having too little time to spend on each game, previews being taken way too seriously, or reviewers simply not understanding the type of game they are testing (my favourite is one reviewer who I think tried to play Europa Universalis III as a "standard" RTS, declared war against a bigger country without building up an army in advance, and then complained about an unbalanced game allowing France to outproduce him:p).

As for the ratings it is rather silly to use a 0-100 scale when a) there is no way to make subjective judgments with that kind of precision (any statistician will be able to tell you that too fine a scale makes for bogus results), b) only the last third of the scale is used. But those are really minor nitpicks when there is a consensus over what the ratings mean. Complaining about that would be like attacking flat earthers not for using the flat earth theory, but for using the imperial system of measurements!
 
Joined
Nov 4, 2006
Messages
2,013
I don't mind 0-100, since a game has so much more stuffs to consider compared to a movie. I think the new gamespot way of rating is very interesting giving the game medals, both good and bad ones it gives you a quick overview and also shows why a game has a certain rating!
 
Joined
Oct 25, 2006
Messages
6,292
I think the author has a good point here:

The fundamental problem with game reviews is that they're analyzing products, not pieces of art. Or more clearly stated, art reviews decide if something is worth your time; game reviews decide if something is worth your money.

about reviewers seeing their job as recommending a purchase instead of analyzing what a game actually is and how it achieves it's goal.

Yes, one of the things you need to know is if it's worth buying, but presenting the game on the same level as a plasma TV or a refrigerator misses the reason why we play games to begin with( the experience) and ends up focusing on appearance, (graphics) and how a game compares to other games instead of what it does that makes it work or not work.
 
Joined
Oct 18, 2006
Messages
7,834
Why not use school grade terms for that ?
 
Joined
Nov 5, 2006
Messages
21,908
Location
Old Europe
I don't think this depends on the rating system. In fact, sites such as metacritic.com use the same rating system in various works. Personally, when I am interested in a work, I read its reviews rather than just numbers, which may allow me to have fewer problems with rating systems. However, I cannot but agree that I am unhappy with the way video games to be evaluated, too. Generally speaking, the writing skills of reviewers of video games don't seem to match those of the reviewers of films/books...
 
Joined
Jul 20, 2007
Messages
278
The article is littered with false premises, broken analogies and unsupported statements.
The fundamental problem with game reviews is that they're analyzing products, not pieces of art. Or more clearly stated, art reviews decide if something is worth your time; game reviews decide if something is worth your money.
It's not a problem, fundamental or otherwise, because:
a) for the most part that's not what game reviews actually do, and
b) even if they did that would be part of the solution not the problem, and
c) it's a false analogy in anycase.

This “games are art” concept is so flawed I don’t even know where to begin. I hope “games are art” advocates don’t go harassing tech support upon encountering game breaking bugs or poor system performance. Remember, art can’t be judged by such uncouth requirements as “fit for purpose” – that’s for tawdry things like consumer products.

Generally speaking, the writing skills of reviewers of video games don't seem to match those of the reviewers of films/books...
Doubtless part of it is lack of technical skill, however they’re also writing for a different audience. A literary review is only of interest to those who are literate, whereas, judging by posts on mainstream forums, the average next-gen gamer is n0t biG on teh r3aDin R thiKin lol omg. I blame SMS messaging, the government, the parents, lack of corporal punishment, and Oblivion.
 
Joined
Feb 8, 2007
Messages
150
Interesting article and I don't really disagree with any of it. But an even bigger problem than trying to find some kind of improved scoring system is the whole 'exclusive' deals many publishers live by.

Exclusivity breeds all kinds of business deals that have a lot more to do with making money than providing an honest review, whatever the scoring system might be.
 
Joined
Oct 18, 2006
Messages
2,897
Location
Oregon
Yeah, I'd be more concerned with the currying of favour between publishers and reviewers.

Even the ostensibly innocent stuff. A writer visits the development team for an article or somesuch. It's easy to be more forgiving or positive when you've had a few laughs with the team, when perhaps you really shouldn't be.
 
Joined
Oct 18, 2006
Messages
122
Location
United Kingdom, London
The real problem with game reviews is that most reviewers simply aren't trustworthy. The scale used for reviews is a very minor problem. Inflated reviews from biased, fanboyish journalists is the reason why my faith in the business is at an all-time low.
 
Joined
Oct 18, 2006
Messages
622
Back
Top Bottom