Gamasutra - The Effect of Reviews on Game Development

magerette

Hedgewitch
Joined
October 18, 2006
Messages
7,834
Gamasutra posts an op-ed from Lewis Pulsipher, former game designer and current teacher of game design, discussing a panel discussion called "Teaching to the Test" from the Triangle Games Conference on the effect of reviews on development of games:
...To explain the title, K12 teachers tend to teach what is on end-of-grade tests to the exclusion of almost everything else. The panel considered how much game development studios and publishers create games to meet the “test” of reviews...
Their answer to the main question was “definitely not,” though they do pay attention to what individual game fans say on forums and email..Benito saw fan opinion as more "pure from the heart" than the reviews...
On the actual review process:
Panelists clearly did not care for reviews in general, probably because they felt so many were poorly-written and often contained mistakes. One panelist specifically referred to the reviews on IGN and Gamespot as “white noise”, and all panelists clearly felt that reviews are often “subjective” rather than “objective”. Of course, “subjective” can be just as accurate (in fact, more accurate) than objective, depending on the situation, the problem is that reviewers don’t explain their biases and why they feel as they do, so readers have no basis to judge the opinions.

Some reviewers clearly don’t understand how reviews, of any medium, work. They should answer three questions:
  • what were the creator(s) trying to do
  • how well did they do it
  • was it worth doing
To answer these questions they must explain “why”, not merely say “this is a piece of junk” or “I don’t like the graphics” or “what a dumb idea”. But this makes reviewing more difficult, more work.
One panelist suggested reviewers ought to “take a step back” and watch others play the game, in order to acquire more than one point of view. They also need to put themselves in the shoes of a person who’s saved his pocket-money to buy a game, as opposed to reviewers who have piles of freebies to try out.
Reviewers who assign an actual numeric evaluation should provide several scores for different types of players, e.g., hard core, casual, RPG fans, shooter fans, whatever is appropriate to the audience.
More information.
 
Joined
Oct 18, 2006
Messages
7,834
I think the problem is that many reviewers do not have (any) knowledge of the different game genres they're reviewing. A person playing most FPS games might be asked to do a review of an adventure game or an rpg like DA: Origins. Of course, there are very good FPS games out there just as there very bad adventure games out there.

The point is here to understand, the difference in the audiences that play say FPS game or adventure games. Recently I played the demo for Secret Files 2; I didn't like it - because it involved a lot of foolish running around to do the most simplest tasks. The puzzles were logical enough, I found; however, the presentation of the puzzles left much to be desired, I've found.

Based on the demo, I would say that only more hardcore adventure player will enjoy this game. And the story started to get interesting at the end of the demo.

If I'm playing a FPS game like Far Cry I can definetely enjoy how the game seems solid in its presentation, its art design, gameplay and appreciate the story in the game. I can see that this is a great game :) --- just like I can see that most adventure players would like to play Secret Files 2. But is may not be something for a newcomer to the adventure game genre - based on the demo at least.

On objective reviews is near impossible, I find. However, the reviewers need at least to write about what game genre it is, how the game is in relations to other games in the same genre. And. of course, about how they had experienced the game when playing the game. They also need to give reasons why they think the game is great, good or 'dumb' or why they don't like the art design at all. If they like a feature in the game, or the gameplay, the reviewers should explain why this is, too....
 
Joined
Oct 18, 2006
Messages
2,147
Location
Denmark, Europe
I think it's easy to fix such an obvious problem as having a reviewer with no experience in the genre of the game he's reviewing. If websites and magazines are really doing that, then they obviously don't give a blankety blank about the quality of their reviews AND nothing anyone could say will change that. However, I think this is not a situation which occurs as frequently as it's complained about. I think it's more often the case that a dev is unhappy with a reviewer who is sufficiently well-versed in a genre, but just plain doesn't like the game's approach to it.

To put a fine point on it, consider CRPGs. If a game reviewer has played every new fairly big-budget CRPG that's come out in the past 5 years, would you say he's familiar with the genre? If his idea of the pinnacle of achievement is Oblivion, or NWN2, or whatever, does that mean this reviewer is incapable of reviewing something like Geneforge? Are we gonig to say that to REALY be familiar with a genre, you have to go back and play 20 years worth of games? Is that necessary to understand and evaluate games that are coming out in the year 2009? Or are we going to split the genre further and further, i.e., "You're familiar with first-person CRPGs a la Fallout 3, but you don't understand isometric ones, so shut up." I don't think so, but I'm a simple, simple man. And where else do we draw the lines? People with real-time backgrounds can't review turn-based games? People familiar with single character games can't review party-based?

Some reviewers clearly don’t understand how reviews, of any medium, work. They should answer three questions:
· what were the creator(s) trying to do
· how well did they do it
· was it worth doing
I think this is a fine paradigm for a reviewer to use, but surely the author must realize that this is just one approach and not THE ONLY approach. Call me old fashioned, but I think the review of video games should describe gameplay in as detailed a manner as possible, and then tell me if the game's fun. We all know that fun's subjective, but I'd certainly like to hear the reviewer's thoughts on the subject. Still, by describing the gameplay and showing me screenshots, I can make the call as to whether I might enjoy what he didn't, or vice versa. Sorry devs, but we gamers don't care the slightest "what the devs were trying to do"; all we care about is how the game turned out. Get over yourselves.
 
Joined
Apr 24, 2007
Messages
374
Location
too poor for Manhattan
...
To put a fine point on it, consider CRPGs. If a game reviewer has played every new fairly big-budget CRPG that's come out in the past 5 years, would you say he's familiar with the genre? If his idea of the pinnacle of achievement is Oblivion, or NWN2, or whatever, does that mean this reviewer is incapable of reviewing something like Geneforge? Are we gonig to say that to REALY be familiar with a genre, you have to go back and play 20 years worth of games? Is that necessary to understand and evaluate games that are coming out in the year 2009? Or are we going to split the genre further and further, i.e., "You're familiar with first-person CRPGs a la Fallout 3, but you don't understand isometric ones, so shut up." I don't think so, but I'm a simple, simple man. And where else do we draw the lines? People with real-time backgrounds can't review turn-based games? People familiar with single character games can't review party-based?
...

I don't think I agree. Maybe I would suck as a reviewer, but I believe it would be unfair for any game FPS or RTS to be reviewed by me, because I simply don't care about those games. I don't see the difference on what makes Farcry fun and Crysis not fun (or viceversa), to me they're all the same. I think the reviewer should be really familiar with at least the game's genre he's reviewing.
Now, the same happens to a degree with sub-genres. I don't really pay much attention to reviewers who are new to the sub-genre. If the reviewer of a turn-based game starts by saying "I've never played a turn based game before" right there it lost me. There are many subtleties and features that are unique to a subgenre that's what makes the game tick. I wouldn't really trust a review of King's Bounty from someone who's only played Diablo. They would usually lower the score if the game doesn't have an online component, or if the stats are too complicated, or that the game is too 'slow'. In fact I've found that, in many cases, what the reviewer complains about is actually what I like about the game.
In the end what I do is read a few reviews, hopefully in places like RPGWatch where people comment on the reviews, try to filter out all comments that are not important to me, and try to derive a personal score from that, trying to ignore the reviewer's score. But sometimes it's not that easy. The reviewer tends to expand on things that he/she particularly liked or disliked, leaving out or just mentioning things that are important to me.
 
Joined
Sep 23, 2008
Messages
5,645
Location
Tardis
A single review is worthless, IMHO. Too much of the reviewer is in the review.

A set of 15 reviews from different sources provide an combined score, on the other hand, is very useful. Inter-reviewer variation is mitigated and the combined score is a helpful metric that can be used to compare titles across genres.

I'll take the benefit of the law of averages over individual scores any day.
 
Joined
May 26, 2008
Messages
59
I'll take the benefit of the law of averages over individual scores any day.

That's the right approach if you want an average result. ;)
 
Joined
Aug 30, 2006
Messages
7,830
The problem with quality is: Who defines what's better?

Should a review get a higher weight only because it was published on Gamespot or IGN?

I remember an interview with that dude from one of the big aggregator sites. He basically said the shown averages are calibrated as he sees fit. It's his job to decide about this.
Very problematic.
 
Joined
Aug 30, 2006
Messages
7,830
That's a cop out. Take responsibility and some initiative.

It's clear to many that many of the new reviewers are incompetent and inexperienced for the genre they are reviewing.

We should:
- review the reviews. Reviewers need to be held to same standards as the games they are reviewing,
- calculate scale factors for all reviews coming from each site/publication based on our evaluation of the reviews from that site,
- apply these scale factors to determine weighted average scores.
- publish these scale factors so readers can be better informed and sites/publications get feedback on how others percieve their reviews.

Simple. ;)
 
Joined
Aug 18, 2008
Messages
15,679
Location
Studio City, CA
Averaging 15 reviews is problematic.

Do you know the 15 reviewers?

To me, the review does little for me if I don't know the reviewer. So I'd like to have seen reviews for multiple games from the same reviewer to really get a feel for his taste.

You might sometimes call a review bad quality when it really comes down to having different opinions of what makes a good game.
 
Joined
Oct 22, 2006
Messages
409
I had no idea game reviewing was so screwed up. If this panel is indicative, then game reviewers see their function as something more like Buyers Lab or Consumer Reports than as entertainment review in the traditional sense. And it seems to have become accepted as a legitimate and valuable function within the game industry. That's fine, but they probably should call it something else.

When I worked in the Performing and Fine Arts department at USC, my boss used to write professional reviews, mostly for opera and ballet. I screened his calls and can tell you his opinion mattered to plenty of folks (which made perfect sense at the time). But he wrote actual reviews, not product assessments.

The reviewer’s job is to give customers information they might want and need in order to make their own decision, like whether or not to attend a show. His job isn’t to cast judgment or provide a ranking. Reviewers should exercise good judgment, of course, but that shouldn’t confuse their function.

This isn’t a subtle point. True reviewers look down their noses (way down) at the Siskel & Ebert’s of the world and their “thumbs up or thumbs down approach.

The current state of game review reflects the current state of game making, IMO. Product quality is so poor that consumer protection has trumped other concerns. The value of subtle appreciation has been trumped by the value of avoiding complete disappointment or even being cheated outright. It’s sad, really.

Oh, and averaging reviews? C'mon, get serious.
 
Joined
Nov 11, 2006
Messages
1,807
Location
Orange County, California
Oh, and averaging reviews? C'mon, get serious.

I am serious.

Let's look at movies. Those that are consistently well-reviewed are gems. Those that receive consistently poor-reviews are not. This is why I trust Amazon/IMDB reviews which are an average of 1,000s of opinions far more than a single review.

It doesn't matter who the reviewers are, high-quality product will always rise to the top in the rankings.
 
Joined
May 26, 2008
Messages
59
It doesn't matter who the reviewers are, high-quality product will always rise to the top in the rankings.
"McDonalds is my kind of place...a hap-hap-hap-happy place...."

Actually, if you're talking about game review in its current state, not in the traditional sense I referred to above, then I might agree with you.
 
Joined
Nov 11, 2006
Messages
1,807
Location
Orange County, California
It is empirically shown that when large number of people guess the answer to a certain question, the mean number given is very close to the truth. Whether you want to apply this to reviews is up to you, but I usually check the Metacritic average and read a few reviews from sources I respect.

When the game is a AAA title, it makes sense to subtract 10 from the average score.
 
Joined
Dec 24, 2008
Messages
557
Location
London, UK
And btw, I don't think the reviewer needs to figure out what the creators tried to do. They need to find out what the *game* is trying to accomplish (specifically, who it is trying to please).
 
Joined
Dec 24, 2008
Messages
557
Location
London, UK
I try to find a reviewer whose opinions regularly line up with mine and stick to what that person has to say. I trust the person, rather than a group of people to tell me whether a game, film, book, whatever is worth my time and money. Sometimes finding a reviewer with whom I always disagree can work almost as well. :)
 
Joined
Aug 31, 2006
Messages
12,806
Location
Australia
Sometimes finding a reviewer with whom I always disagree can work almost as well. :)

Lol Corwin. Right on indeed ;-) Works for "word of mouth" too: had a friend in high school, every time he recommended a video game or a movie to me, I knew I wasn't going to like it. So he was actually the best "shopping guide" I've ever had. Ironically, we had the exact same tastes in music and sports though, and that's probably what cemented our friendship.

As for reviews, I've found that "time" often correlates with "quality". When reviews are published within one week of release, I'm often disappointed. Contrarily, I've really been enjoying Eurogamer's re-reviews and RPGWatch "late" reviews, such as your and txa's review of Fallout which came in late January.
 
Joined
Dec 24, 2006
Messages
203
Wait till you read our late review of Drakensang!! :) Oops, that reminds me, perhaps I'd better write one!!!! :D
 
Joined
Aug 31, 2006
Messages
12,806
Location
Australia
As a teacher, I tend to overanalyze things too much maybe ;)

That said, however, I do feel that one can use some of the same tecniques in analyzing a video game, mapping out its themes, understand the characters and the dialogue in the game as you would do when you analyzed a book, a movie or a tv-show or even a play.

I don't need to see reviewer rave on about how the game got the geography of Paris of Paris wrong or how he or she didn't like the French voiceacting (like a recent review of adventure game, I've read) or that the reviewer don't like the D&D genre at all.

I need the reviewer to stay focused and tell me about features that are in the game, how is the game's dialogue, the characters in the game, how is the story progressing througout the game, are the puzzles in the gamelogical or not (if it is an adventure game) or how is the combat (if it is an rpg or an fps game) and of course, I'll also need the reviewer to be given his or her subjective attitude to the game, but based on the game's own merit - not who the author was or the company that released it. And by that I mean, just because you (as in the general you) like Bioware, Bioware's games should not be let off the hook easier than others.

I also agree that the whole 'get it later-get it better' feature works for me :)
However, it may only work for sites like these and eurogamer since they(we) don't have to earn any money. A newspaper reviewing a game has not such privilege, they just need to get the reviews out when the game is released.
 
Joined
Oct 18, 2006
Messages
2,147
Location
Denmark, Europe
I am serious.

Let's look at movies. Those that are consistently well-reviewed are gems. Those that receive consistently poor-reviews are not. This is why I trust Amazon/IMDB reviews which are an average of 1,000s of opinions far more than a single review.

It doesn't matter who the reviewers are, high-quality product will always rise to the top in the rankings.
The Shawshank Redemption could never happen in videogaming.
 
Joined
Mar 30, 2008
Messages
1,163
Location
Scandinavia
Back
Top Bottom