Metacritic Matters- How Review Scores Hurt Video Games

Couchpotato

Part-Time News-bot
Joined
October 1, 2010
Messages
36,194
Location
Spudlandia
Kotaku brings an article about Metacritic and how they affect video games.
Bugs in Fallout: New Vegas might have eaten your save file. Maybe they took away a few hours of progress, or forced you to reset a couple of quests. Maybe game-crashing bugs pissed you off to the point where you wished you could get your $60 back. But they probably didn’t cost you a million dollars.

Perhaps you've heard the story: publisher Bethesda was due to give developer Obsidian a bonus if their post-apocalyptic RPG averaged an 85 on Metacritic, the review aggregation site. It got an 84 on PC and Xbox 360, and an 82 on PS3.

“If only it was a stable product and didn't ship with so many bugs, I would've given New Vegas a higher score,” wrote a reviewer for the website 1up, which gave New Vegas a B, or 75 on Metacritic's scale.

“It's disappointing to see such an otherwise brilliant and polished game suffer from years-old bugs, and unfortunately our review score for the game has to reflect that,” said The Escapist's review, which gave the game an 80.

If New Vegas had hit an 85, Obsidian would have gotten their bonus. And according to one person familiar with the situation who asked not to be named while speaking to Kotaku, that bonus was worth $1 million. For a team of 70 or so, that averages out to around $14,000 a person. Enough for a cheap car. Maybe a few mortgage payments.

Those sure were some costly bugs.

Another problem for developers: outlier scores. What happens when tons of people like a game, but for one or two reviewers, it just doesn’t click?

“The problem is the scale,” said Obsidian’s Urquhart. “There's an expectation that a good game is between 80 and 90. If a good game is between 80 and 90, and let's say an average game is gonna maybe get 50 scores, if you wanna hit that 85 and someone gives you a 35, that just took ten 90s down to 85… Just math-wise, how do you deal with that? Some guy who wants to make a name for himself can absolutely screw the numbers.”
There's plenty of more information in the article. Now do you Agree or Disagree?
More information.
 
Last edited:
Joined
Oct 1, 2010
Messages
36,194
Location
Spudlandia
Well, that's why some people do not use mean but median, which is an unbiased estimator, and much more robust and less sensitive to outliers. If the population can be approximated with a Gaussian, then it doesn't matter which one you use. If it is log-Gaussian, then one should use geometric mean anyways.
Maybe Metacritic should switch to that.

edit: Just read the article, and their system seems much more complicated with all kinds of weighting and what-not. But they still should be able to come up with a decent statistical analysis that's not too sensitive to outliers.
 
Last edited:
Joined
Sep 1, 2007
Messages
29
Well given how SHITTY Sim City (5) runs, I CAN ONLY HOPE it hurts them bad!
 
Joined
Aug 24, 2007
Messages
555
Location
Germany
Is Uncle Feargus implying that buying reviews isn't enough to get you anywhere because one renegade reviewer can drag you down?
 
Seems like some reviewers will slag a game regardless of its quality. They think being a critic means you must find fault with whatever you review. And some reviewers are completely inexperienced with the game under review. Legends in their own mind, if you will.
 
Joined
Jan 23, 2009
Messages
601
Location
Minnesota
Shouldn't bonuses be based off of sales and not some idotic media score? I don't know the last time i actually looked at a metacritic score. That means nothing to me.
 
Joined
Nov 11, 2006
Messages
713
Seems like some reviewers will slag a game regardless of its quality. They think being a critic means you must find fault with whatever you review. And some reviewers are completely inexperienced with the game under review. Legends in their own mind, if you will.

And also giving a near-universally popular game a very low score can drive incredible amounts of traffic to your site. Maybe not as fun as inciting flame wars between over zealous fans of various games with inane articles arguing, just to give a ridiculous hypothetical example, how Call of Duty 4 is an objectively superior game to Bioshock Infinite because your criteria for what constitutes a good game or even sound game design produces a scale that kind of sounds like it goes from 0 to Call of duty. I'm kind of surprised an IGN editor hasn't done that one yet as an op ED/dissenting review opnion.
 
Joined
Nov 20, 2006
Messages
1,710
The article starts with a logical fallacy:

Personally, I wish I had trusted in the negative Fallout: New Vegas reviews, before I wasted many an hour because of broken save game files that finally prevented me from progress even after repeating large parts of the game. At some point, each load would end in a crash to desktop, despite the cautious approach of starting the game several months after release.

The game was bought by millions, and if only several percent experienced the same bugs, multiplying the frustration, the lost time and money by the number of people affected does justify a loss of millions for the developer.

I like Obsidian, NWN2: Mask of the Betrayer still rates as one of my favourite games, but this loss was well deserved.
 
Joined
Oct 18, 2006
Messages
471
The game was bought by millions, and if only several percent experienced the same bugs, multiplying the frustration, the lost time and money by the number of people affected does justify a loss of millions for the developer.

I like Obsidian, NWN2: Mask of the Betrayer still rates as one of my favourite games, but this loss was well deserved.

I find this narrow minded and that is why I normally ignore user scores and focus on major reviewer scores (for better or worse). I played F:NV one month after release for more than 100 hours and while it was patched without a single crash on a 5 year old hardware. Bugs in RPGs are common particularly for games of similar scale to F:NV, and their patching takes a while and downgrading them because angry Joe does not know or care how to make multiple saves and had a couple of crashes on their PC (which PC gamers are accustomed to by nature of this platform and associated software and hardware) is childish. As for major reviewers, I wonder why they did not bash Skyrim in the same way as they did F:NV?
 
Joined
Feb 2, 2011
Messages
2,818
Location
United Kingdom
I have to admit, I don't think I've read a single review beforehand when I've purchased a game in the last 25 yrs or so. I do, however, listen to a few friends and that's directly responsible to my buying games 90% of the time, I'd guess. I was warned about FNV so I held off buying that one for about 2 weeks, but I did preorder Skyrim and got burned there. I'm more interested in the truth, and be it a lemon game or car or house, I've no problem at all with people honestly bashing lemons. It does, however, need to be honest.



-Carn
 
Joined
Oct 18, 2011
Messages
18,808
Location
Holly Hill, FL.
I find this narrow minded and that is why I normally ignore user scores and focus on major reviewer scores (for better or worse). I played F:NV one month after release for more than 100 hours and while it was patched without a single crash on a 5 year old hardware. Bugs in RPGs are common particularly for games of similar scale to F:NV, and their patching takes a while and downgrading them because angry Joe does not know or care how to make multiple saves and had a couple of crashes on their PC (which PC gamers are accustomed to by nature of this platform and associated software and hardware) is childish. As for major reviewers, I wonder why they did not bash Skyrim in the same way as they did F:NV?

Insulting me is not necessary. You were lucky, the game worked for you. For some people, it did not. Will you just dismiss their opinion because you did not share in their experience?

Just for the record, I did have multiple saves and tried reverting to an earlier one that worked, initially. I accepted repeating many hours of the game because of this, but eventually the same crashes reappeared without obvious cause. I read the message boards: no solution and many reports of similar problems. Any other software ran fine on my system, which was nothing peculiar either.

Anyway, my point is different: if you owe a 100,000 people 10 dollar each, you are a million in debt, not 10 dollar. The article's initial comparison of 60$ lost by an individual versus millions of lost revenue for Obsidian is not valid.

Also, I am a scientist heavily involved in software engineering, and was not particularly angry: just frustrated.
 
Joined
Oct 18, 2006
Messages
471
The article lacks a little perspective. I'm going to pull a tiny bit of self-indulged rank here - I've been gaming since the original Odyssey videogame console. The only people more nerdy than me are the pioneers who created all this stuff.

Since that time, all the way up to the invention of Metacritic, there hasn't been a single thing to organize a 'voice' for gamers nearly as effectively as Metacritic. It's not perfect. And I've disagreed with some Metacritic scores I've seen for both movies and videogames. But I wouldn't want it to go away.

I don't expect perfect, bug free games from videogames. But as a matter of routine, publishers are guilty of rushing things out the door to meet deadlines at the expense of gamer's wallets in the form of game breaking bugs. We all pay with currency that is useable and not broken but in return so many times we get products that don't work or are so glitchy as to suck any potential enjoyment away.

Metacritic has finally applied very effective pressure to the only thing publishers will listen to - which is the bottom line. Publishers ignored gamers for nearly 30 years when there was no objective data that actually affected sales.

Metacritic may not be perfect, but I don't want to see the pressure relieved. Even now with Metacritic games continue to be released in time for the almighty marketing release date and not when they've been cooked enough to actually work.

I'm sure publishers would love to see changes with the way Metacritic computes scores so that everyone's game gets 100. While publishers are the mechanism games traditionally reach the marketplace, they are also responsible for buggy and 'samey' gameplay. So a big thanks to Metacritic. And also a big thanks to the new kid on the block, kickstarter.
 
Joined
Oct 18, 2006
Messages
2,897
Location
Oregon
Trying to address this in unbiased a way as possible, I would say that:
1) Bethesda's game engines are a joke commonly glossed over with PR management and money greasing the wheels. To blame Obsidian for its shoddiness is misdirected.
2) Metacritic is a farce and should have never been used as a lynchpin in determining a bonus.

Nevertheless, reviews are reviews. And bugs are bugs. Oh well.
 
Joined
Nov 10, 2008
Messages
5,979
Location
Florida, USA
This reminds me of a previous job I had. I was a trainer, and part of my bonus each month depended on my review scores. They would send out grading sheets to my trainees after the training was done and ask them to rate my performance. Most of it was good but occasionally you would get some renegade trainee who decided to rate me poorly, even though I gave the same quality training to that person as I did to the others. One poor review could ruin my bonus, so you just had to hope that didn't happen to you.

Of course, my bonus wasn't a million dollars, but I can see the same thing happening to them that happened to me. Sucks but that's just how it goes.
 
The article starts with a logical fallacy:

Personally, I wish I had trusted in the negative Fallout: New Vegas reviews, before I wasted many an hour because of broken save game files that finally prevented me from progress even after repeating large parts of the game. At some point, each load would end in a crash to desktop, despite the cautious approach of starting the game several months after release.

The game was bought by millions, and if only several percent experienced the same bugs, multiplying the frustration, the lost time and money by the number of people affected does justify a loss of millions for the developer.

I like Obsidian, NWN2: Mask of the Betrayer still rates as one of my favourite games, but this loss was well deserved.

Has the issue you had ever been fixed? I was thinking about the problem you had with save game corruption and since it was only happening with a few people it may not be the game that is causing it but rather system corruption that could be causing interference with the game. In all of the years I have been repairing computers that is the most common issue I have run into which required a repair or reinstall of windows. All games interact with the OS on some level so if there is corruption in the area that the game interacts with then it can cause issues in the game.
 
Joined
Oct 19, 2006
Messages
1,596
One question I ask myself :

How do people look at these shitstorms in 20-30 years ?

The replies will still be there - let's say on Amazon - but the game might already be gone - how will people be able to verify that such an shitstorm was actually right (or wrong) in 20 or in 30 years ?

But that just doesn't matter to those who drive shotstorms on . All that matters to them is their current pain.

In 20 years their pain might be gone - and : "what ? What did I write there ?"
Or they totally forgot their "hit & run fire & forget" account only made for posting an review expressing their anger ...


Another point I really ask myself is this : Why shouldn't honest players not be allowed to post their negative reviews ? - As long as they state that "this game isn't for them", of course. There are more than enough book critics, for example Mr. Reich-Ranicki, who is infamous for his harsh critics.
 
Joined
Nov 5, 2006
Messages
21,915
Location
Old Europe
When it comes to buying games, I rarely heed reviews from critics. My method is to watch their development, view the trailers, and after release I watch real game-play on YouTube. Even if a real player gives a 'thumbs down' on YouTube, I still might get the game just because it appeals to me and I like that particular game's playability, look. or genre. I guess I tend to like what I like regardless of what anyone thinks.
 
Joined
Jan 23, 2009
Messages
601
Location
Minnesota
But that just doesn't matter to those who drive shotstorms on . All that matters to them is their current pain.

As a consumer, I've found it greatly useful to read user reviews about products. What I've found over the last 10 years is that you cannot rely on any one user review to base a purchase decision on.

I look for trends. I look for particular comments that are echoed again and again by many different people across at least several different review sites.

If you look at the user comments on metacritic for Diablo 3, there are certainly plenty of posts that meet your 'shitstorm' description.

But what validates those users' inflammatory comments is that the same issues are cited again and again by many people, not just one or two or a small percentage relative to a whole.

You could give away a pot of gold to 100 people and at least a couple of them will post on the internet that the experience caused them great amounts of grief and that I was an asshole to give them a pot of gold.

It's the trends that can at least give one a chance at knowing some truth about a product. That's why I like metacritic - when something is terribly wrong with a product, there will be just way too many people all saying the same things. They could all be wrong. But when it's an overwhelming number of complaints all centered around the same issues, they usually aren't wrong.
 
Joined
Oct 18, 2006
Messages
2,897
Location
Oregon
I find this narrow minded and that is why I normally ignore user scores and focus on major reviewer scores (for better or worse). I played F:NV one month after release for more than 100 hours and while it was patched without a single crash on a 5 year old hardware. . . . .

I'm sorta the opposite. Critic reviews might get me interest me in a game, but when it comes to a game that I am on the fence for, user reviews are what sways me. (See Divine Divinity)

BTW, Fallout Vegas was one of the best RPGs I had ever played but I couldn't go more than 2 hours without a crash that stopped my machine. Not a crash to desktop but a crash that crashed the machine. I had a ton of problems with that game but it was so good, I just put up with it.
 
Joined
Oct 19, 2006
Messages
1,762
Location
Los Angeles area
Has the issue you had ever been fixed? I was thinking about the problem you had with save game corruption and since it was only happening with a few people it may not be the game that is causing it but rather system corruption that could be causing interference with the game. In all of the years I have been repairing computers that is the most common issue I have run into which required a repair or reinstall of windows. All games interact with the OS on some level so if there is corruption in the area that the game interacts with then it can cause issues in the game.

I did not try to run Fallout: NV again, but the Windows installation is still the same I am running now and the software part was working well continuously. There was a hardware problem manifesting in a BSOD with a DDR3 DIMM in February this year, which required re-plugging the module before memtest86+ ran without error again. In theory, this might have caused save game corruption if the error had persisted unnoticed all the time from 2011 to early 2013. It is not likely the cause, though, since no other file damage was caused from this on both operating systems, Windows 7 and Ubuntu: checksums work out.

Drithius has a good point regarding Bethesda's engines: they are infamous for unpredictable behaviour and probably a mess to handle for a complex task like F3:NV. On the other hand, Fallout 3 itself worked fine for me. Also, I am sure Obsidian would not shift blame to their set of tools themselves: it seems inappropriate. Better to accept the mistake, take the losses and learn from the experience.

I find myself agreeing with TheMadGamer's post on the subject. Metacritic is not perfect, but serves a function. I like to have user opinions as well: there will be some astroturfing usually, but reading between the lines is still possible. Professional reviews are often not that useful to me.
 
Joined
Oct 18, 2006
Messages
471
Back
Top Bottom