|
Your continuous donations keep RPGWatch running!
RPGWatch Forums » Comments » News Comments » Gamasutra - The Effect of Reviews on Game Development

Default Gamasutra - The Effect of Reviews on Game Development

May 5th, 2009, 19:24
Gamasutra posts an op-ed from Lewis Pulsipher, former game designer and current teacher of game design, discussing a panel discussion called "Teaching to the Test" from the Triangle Games Conference on the effect of reviews on development of games:
…To explain the title, K12 teachers tend to teach what is on end-of-grade tests to the exclusion of almost everything else. The panel considered how much game development studios and publishers create games to meet the “test” of reviews…
Their answer to the main question was “definitely not,” though they do pay attention to what individual game fans say on forums and email..Benito saw fan opinion as more "pure from the heart" than the reviews…
On the actual review process:
Panelists clearly did not care for reviews in general, probably because they felt so many were poorly-written and often contained mistakes. One panelist specifically referred to the reviews on IGN and Gamespot as “white noise”, and all panelists clearly felt that reviews are often “subjective” rather than “objective”. Of course, “subjective” can be just as accurate (in fact, more accurate) than objective, depending on the situation, the problem is that reviewers don’t explain their biases and why they feel as they do, so readers have no basis to judge the opinions.

Some reviewers clearly don’t understand how reviews, of any medium, work. They should answer three questions:
  • what were the creator(s) trying to do
  • how well did they do it
  • was it worth doing
To answer these questions they must explain “why”, not merely say “this is a piece of junk” or “I don’t like the graphics” or “what a dumb idea”. But this makes reviewing more difficult, more work.
One panelist suggested reviewers ought to “take a step back” and watch others play the game, in order to acquire more than one point of view. They also need to put themselves in the shoes of a person who’s saved his pocket-money to buy a game, as opposed to reviewers who have piles of freebies to try out.
Reviewers who assign an actual numeric evaluation should provide several scores for different types of players, e.g., hard core, casual, RPG fans, shooter fans, whatever is appropriate to the audience.
More information.

Where there's smoke, there's mirrors.
magerette is offline

magerette

magerette's Avatar
Hedgewitch
RPGWatch Team

#1

Join Date: Oct 2006
Posts: 7,929

Default 

May 5th, 2009, 19:24
I think the problem is that many reviewers do not have (any) knowledge of the different game genres they're reviewing. A person playing most FPS games might be asked to do a review of an adventure game or an rpg like DA: Origins. Of course, there are very good FPS games out there just as there very bad adventure games out there.

The point is here to understand, the difference in the audiences that play say FPS game or adventure games. Recently I played the demo for Secret Files 2; I didn't like it - because it involved a lot of foolish running around to do the most simplest tasks. The puzzles were logical enough, I found; however, the presentation of the puzzles left much to be desired, I've found.

Based on the demo, I would say that only more hardcore adventure player will enjoy this game. And the story started to get interesting at the end of the demo.

If I'm playing a FPS game like Far Cry I can definetely enjoy how the game seems solid in its presentation, its art design, gameplay and appreciate the story in the game. I can see that this is a great game —- just like I can see that most adventure players would like to play Secret Files 2. But is may not be something for a newcomer to the adventure game genre - based on the demo at least.

On objective reviews is near impossible, I find. However, the reviewers need at least to write about what game genre it is, how the game is in relations to other games in the same genre. And. of course, about how they had experienced the game when playing the game. They also need to give reasons why they think the game is great, good or 'dumb' or why they don't like the art design at all. If they like a feature in the game, or the gameplay, the reviewers should explain why this is, too….

Please support http://www.maternityworldwide.org/ - and save a mother giving birth to a child.
aries100 is offline

aries100

SasqWatch
RPGWatch Team RPGWatch Donor

#2

Join Date: Oct 2006
Location: Denmark, Europe
Posts: 2,010

Default 

May 5th, 2009, 20:36
I think it's easy to fix such an obvious problem as having a reviewer with no experience in the genre of the game he's reviewing. If websites and magazines are really doing that, then they obviously don't give a blankety blank about the quality of their reviews AND nothing anyone could say will change that. However, I think this is not a situation which occurs as frequently as it's complained about. I think it's more often the case that a dev is unhappy with a reviewer who is sufficiently well-versed in a genre, but just plain doesn't like the game's approach to it.

To put a fine point on it, consider CRPGs. If a game reviewer has played every new fairly big-budget CRPG that's come out in the past 5 years, would you say he's familiar with the genre? If his idea of the pinnacle of achievement is Oblivion, or NWN2, or whatever, does that mean this reviewer is incapable of reviewing something like Geneforge? Are we gonig to say that to REALY be familiar with a genre, you have to go back and play 20 years worth of games? Is that necessary to understand and evaluate games that are coming out in the year 2009? Or are we going to split the genre further and further, i.e., "You're familiar with first-person CRPGs a la Fallout 3, but you don't understand isometric ones, so shut up." I don't think so, but I'm a simple, simple man. And where else do we draw the lines? People with real-time backgrounds can't review turn-based games? People familiar with single character games can't review party-based?

Some reviewers clearly don’t understand how reviews, of any medium, work. They should answer three questions:
· what were the creator(s) trying to do
· how well did they do it
· was it worth doing
I think this is a fine paradigm for a reviewer to use, but surely the author must realize that this is just one approach and not THE ONLY approach. Call me old fashioned, but I think the review of video games should describe gameplay in as detailed a manner as possible, and then tell me if the game's fun. We all know that fun's subjective, but I'd certainly like to hear the reviewer's thoughts on the subject. Still, by describing the gameplay and showing me screenshots, I can make the call as to whether I might enjoy what he didn't, or vice versa. Sorry devs, but we gamers don't care the slightest "what the devs were trying to do"; all we care about is how the game turned out. Get over yourselves.
Yeesh is offline

Yeesh

Yeesh's Avatar
Hopeless Curmudgeon

#3

Join Date: Apr 2007
Location: too poor for Manhattan
Posts: 374

Default 

May 5th, 2009, 21:01
Originally Posted by Yeesh View Post

To put a fine point on it, consider CRPGs. If a game reviewer has played every new fairly big-budget CRPG that's come out in the past 5 years, would you say he's familiar with the genre? If his idea of the pinnacle of achievement is Oblivion, or NWN2, or whatever, does that mean this reviewer is incapable of reviewing something like Geneforge? Are we gonig to say that to REALY be familiar with a genre, you have to go back and play 20 years worth of games? Is that necessary to understand and evaluate games that are coming out in the year 2009? Or are we going to split the genre further and further, i.e., "You're familiar with first-person CRPGs a la Fallout 3, but you don't understand isometric ones, so shut up." I don't think so, but I'm a simple, simple man. And where else do we draw the lines? People with real-time backgrounds can't review turn-based games? People familiar with single character games can't review party-based?
I don't think I agree. Maybe I would suck as a reviewer, but I believe it would be unfair for any game FPS or RTS to be reviewed by me, because I simply don't care about those games. I don't see the difference on what makes Farcry fun and Crysis not fun (or viceversa), to me they're all the same. I think the reviewer should be really familiar with at least the game's genre he's reviewing.
Now, the same happens to a degree with sub-genres. I don't really pay much attention to reviewers who are new to the sub-genre. If the reviewer of a turn-based game starts by saying "I've never played a turn based game before" right there it lost me. There are many subtleties and features that are unique to a subgenre that's what makes the game tick. I wouldn't really trust a review of King's Bounty from someone who's only played Diablo. They would usually lower the score if the game doesn't have an online component, or if the stats are too complicated, or that the game is too 'slow'. In fact I've found that, in many cases, what the reviewer complains about is actually what I like about the game.
In the end what I do is read a few reviews, hopefully in places like RPGWatch where people comment on the reviews, try to filter out all comments that are not important to me, and try to derive a personal score from that, trying to ignore the reviewer's score. But sometimes it's not that easy. The reviewer tends to expand on things that he/she particularly liked or disliked, leaving out or just mentioning things that are important to me.
wolfing is offline

wolfing

wolfing's Avatar
Wonders what SasqWatch is

#4

Join Date: Sep 2008
Posts: 3,207

Default 

May 5th, 2009, 21:03
A single review is worthless, IMHO. Too much of the reviewer is in the review.

A set of 15 reviews from different sources provide an combined score, on the other hand, is very useful. Inter-reviewer variation is mitigated and the combined score is a helpful metric that can be used to compare titles across genres.

I'll take the benefit of the law of averages over individual scores any day.
martink is offline

martink

Watcher

#5

Join Date: May 2008
Posts: 59

Default 

May 5th, 2009, 21:06
Originally Posted by martink View Post
I'll take the benefit of the law of averages over individual scores any day.
That's the right approach if you want an average result.
Gorath is offline

Gorath

Gorath's Avatar
Prime Evil
RPGWatch Team

#6

Join Date: Aug 2006
Posts: 6,818

Default 

May 5th, 2009, 21:12
No. An average of trash is still trash.

Quality reviews over quantity.
Thrasher is offline

Thrasher

Thrasher's Avatar
Wheeee!
RPGWatch Donor

#7

Join Date: Aug 2008
Location: Studio City, CA
Posts: 9,972

Default 

May 5th, 2009, 21:26
The problem with quality is: Who defines what's better?

Should a review get a higher weight only because it was published on Gamespot or IGN?

I remember an interview with that dude from one of the big aggregator sites. He basically said the shown averages are calibrated as he sees fit. It's his job to decide about this.
Very problematic.
Gorath is offline

Gorath

Gorath's Avatar
Prime Evil
RPGWatch Team

#8

Join Date: Aug 2006
Posts: 6,818

Default 

May 5th, 2009, 21:41
That's a cop out. Take responsibility and some initiative.

It's clear to many that many of the new reviewers are incompetent and inexperienced for the genre they are reviewing.

We should:
- review the reviews. Reviewers need to be held to same standards as the games they are reviewing,
- calculate scale factors for all reviews coming from each site/publication based on our evaluation of the reviews from that site,
- apply these scale factors to determine weighted average scores.
- publish these scale factors so readers can be better informed and sites/publications get feedback on how others percieve their reviews.

Simple.
Thrasher is offline

Thrasher

Thrasher's Avatar
Wheeee!
RPGWatch Donor

#9

Join Date: Aug 2008
Location: Studio City, CA
Posts: 9,972

Default 

May 5th, 2009, 22:02
Averaging 15 reviews is problematic.

Do you know the 15 reviewers?

To me, the review does little for me if I don't know the reviewer. So I'd like to have seen reviews for multiple games from the same reviewer to really get a feel for his taste.

You might sometimes call a review bad quality when it really comes down to having different opinions of what makes a good game.
JuliusMagnus is offline

JuliusMagnus

Indulgent Wastrel

#10

Join Date: Oct 2006
Posts: 404

Default 

May 5th, 2009, 22:26
I had no idea game reviewing was so screwed up. If this panel is indicative, then game reviewers see their function as something more like Buyers Lab or Consumer Reports than as entertainment review in the traditional sense. And it seems to have become accepted as a legitimate and valuable function within the game industry. That's fine, but they probably should call it something else.

When I worked in the Performing and Fine Arts department at USC, my boss used to write professional reviews, mostly for opera and ballet. I screened his calls and can tell you his opinion mattered to plenty of folks (which made perfect sense at the time). But he wrote actual reviews, not product assessments.

The reviewer’s job is to give customers information they might want and need in order to make their own decision, like whether or not to attend a show. His job isn’t to cast judgment or provide a ranking. Reviewers should exercise good judgment, of course, but that shouldn’t confuse their function.

This isn’t a subtle point. True reviewers look down their noses (way down) at the Siskel & Ebert’s of the world and their “thumbs up or thumbs down approach.

The current state of game review reflects the current state of game making, IMO. Product quality is so poor that consumer protection has trumped other concerns. The value of subtle appreciation has been trumped by the value of avoiding complete disappointment or even being cheated outright. It’s sad, really.

Oh, and averaging reviews? C'mon, get serious.

Oh, I wish I had a river I could skate away on. But it don't snow here. It stays pretty green. I'm going to make a lot of money, then I'm going to quit this crazy scene. — [Joni Mitchell]
Squeek is offline

Squeek

Squeek's Avatar
connoisseur of tidbits

#11

Join Date: Nov 2006
Location: Orange County, California
Posts: 1,807

Default 

May 5th, 2009, 22:34
Originally Posted by Squeek View Post
Oh, and averaging reviews? C'mon, get serious.
I am serious.

Let's look at movies. Those that are consistently well-reviewed are gems. Those that receive consistently poor-reviews are not. This is why I trust Amazon/IMDB reviews which are an average of 1,000s of opinions far more than a single review.

It doesn't matter who the reviewers are, high-quality product will always rise to the top in the rankings.
martink is offline

martink

Watcher

#12

Join Date: May 2008
Posts: 59

Default 

May 5th, 2009, 22:40
Originally Posted by martink View Post
It doesn't matter who the reviewers are, high-quality product will always rise to the top in the rankings.
"McDonalds is my kind of place…a hap-hap-hap-happy place…."

Actually, if you're talking about game review in its current state, not in the traditional sense I referred to above, then I might agree with you.

Oh, I wish I had a river I could skate away on. But it don't snow here. It stays pretty green. I'm going to make a lot of money, then I'm going to quit this crazy scene. — [Joni Mitchell]
Squeek is offline

Squeek

Squeek's Avatar
connoisseur of tidbits

#13

Join Date: Nov 2006
Location: Orange County, California
Posts: 1,807

Default 

May 5th, 2009, 23:17
It is empirically shown that when large number of people guess the answer to a certain question, the mean number given is very close to the truth. Whether you want to apply this to reviews is up to you, but I usually check the Metacritic average and read a few reviews from sources I respect.

When the game is a AAA title, it makes sense to subtract 10 from the average score.
Grandor Dragon is offline

Grandor Dragon

SasqWatch

#14

Join Date: Dec 2008
Location: Ravensburg, Germany
Posts: 554

Default 

May 5th, 2009, 23:38
And btw, I don't think the reviewer needs to figure out what the creators tried to do. They need to find out what the *game* is trying to accomplish (specifically, who it is trying to please).
Grandor Dragon is offline

Grandor Dragon

SasqWatch

#15

Join Date: Dec 2008
Location: Ravensburg, Germany
Posts: 554

Default 

May 6th, 2009, 00:04
I try to find a reviewer whose opinions regularly line up with mine and stick to what that person has to say. I trust the person, rather than a group of people to tell me whether a game, film, book, whatever is worth my time and money. Sometimes finding a reviewer with whom I always disagree can work almost as well.

If God said it, then that settles it!!

Editor@RPGWatch
Corwin is offline

Corwin

Corwin's Avatar
On The Razorblade of Life
RPGWatch Team

#16

Join Date: Aug 2006
Location: Australia
Posts: 10,405
Send a message via Skype™ to Corwin

Default 

May 6th, 2009, 01:09
Originally Posted by Corwin View Post
Sometimes finding a reviewer with whom I always disagree can work almost as well.
Lol Corwin. Right on indeed ;-) Works for "word of mouth" too: had a friend in high school, every time he recommended a video game or a movie to me, I knew I wasn't going to like it. So he was actually the best "shopping guide" I've ever had. Ironically, we had the exact same tastes in music and sports though, and that's probably what cemented our friendship.

As for reviews, I've found that "time" often correlates with "quality". When reviews are published within one week of release, I'm often disappointed. Contrarily, I've really been enjoying Eurogamer's re-reviews and RPGWatch "late" reviews, such as your and txa's review of Fallout which came in late January.
Hedek is offline

Hedek

Hedek's Avatar
Watchdog

#17

Join Date: Dec 2006
Posts: 203

Default 

May 6th, 2009, 06:04
Wait till you read our late review of Drakensang!! Oops, that reminds me, perhaps I'd better write one!!!!

If God said it, then that settles it!!

Editor@RPGWatch
Corwin is offline

Corwin

Corwin's Avatar
On The Razorblade of Life
RPGWatch Team

#18

Join Date: Aug 2006
Location: Australia
Posts: 10,405
Send a message via Skype™ to Corwin

Default 

May 6th, 2009, 10:16
As a teacher, I tend to overanalyze things too much maybe

That said, however, I do feel that one can use some of the same tecniques in analyzing a video game, mapping out its themes, understand the characters and the dialogue in the game as you would do when you analyzed a book, a movie or a tv-show or even a play.

I don't need to see reviewer rave on about how the game got the geography of Paris of Paris wrong or how he or she didn't like the French voiceacting (like a recent review of adventure game, I've read) or that the reviewer don't like the D&D genre at all.

I need the reviewer to stay focused and tell me about features that are in the game, how is the game's dialogue, the characters in the game, how is the story progressing througout the game, are the puzzles in the gamelogical or not (if it is an adventure game) or how is the combat (if it is an rpg or an fps game) and of course, I'll also need the reviewer to be given his or her subjective attitude to the game, but based on the game's own merit - not who the author was or the company that released it. And by that I mean, just because you (as in the general you) like Bioware, Bioware's games should not be let off the hook easier than others.

I also agree that the whole 'get it later-get it better' feature works for me
However, it may only work for sites like these and eurogamer since they(we) don't have to earn any money. A newspaper reviewing a game has not such privilege, they just need to get the reviews out when the game is released.

Please support http://www.maternityworldwide.org/ - and save a mother giving birth to a child.
aries100 is offline

aries100

SasqWatch
RPGWatch Team RPGWatch Donor

#19

Join Date: Oct 2006
Location: Denmark, Europe
Posts: 2,010

Default 

May 6th, 2009, 10:26
Originally Posted by martink View Post
I am serious.

Let's look at movies. Those that are consistently well-reviewed are gems. Those that receive consistently poor-reviews are not. This is why I trust Amazon/IMDB reviews which are an average of 1,000s of opinions far more than a single review.

It doesn't matter who the reviewers are, high-quality product will always rise to the top in the rankings.
The Shawshank Redemption could never happen in videogaming.
hishadow is offline

hishadow

Level N+1

#20

Join Date: Mar 2008
Location: Southern parts of Norway
Posts: 1,140
RPGWatch Forums » Comments » News Comments » Gamasutra - The Effect of Reviews on Game Development
Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 18:51.
Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
Copyright by RPGWatch