The context and long-term relevance of scores as a means to refence quality

Nereida said:

Alright, just needed to make you aware that if you take the, say, 50,000 people who left a review on steam and then the 20 people who voted in a poll about Neoverse in a Neoverse fan site, and averaged them on a 1:1 proportion, sure as hell Neoverse will come up as the best RPG of all time, so the data has no scientific value.

It is still a beautiful exercise, and I can tell you are enjoying it, so for what is worth, I'm happy to see what you come up with, if not as evidence for anything, just to see the contrast between the general taste of the collective of RPG players, and the biased taste of a selected few.

Click to expand...

Fascinating how you're now using population as your primary argument.

I could have sworn you said earlier that population arguments were laughably irrelevant.

Further, this site isn't a 'specific game' fan-site. It's also not a general gaming fan-site (like others you've mentioned). It's a GENRE fan-site. A genre fan-site with a broad and representative cross-section of the entire genre.

There is no better place to get an idea of what RPG fans think of a game than here. Whatever the number of votes.

Meanwhile, I made this other statistic pane. It is valid because I didn't vote in Steam, and so it shows a perfectly scientifically accurate value.

Since I have a high opinion of myself and I arbitrarily decided that me and my buddies (but mostly me) are the ones who knows most about RPGs and hold the absolute rule on what is the quality of them, there is no better place to look at than averaging my own opinion in a 1:1 proportion against the opinions of thousands, for an accurate statistical value.

Another point you made earlier was that the RPGWatch vote didn't differentiate quality, merely whether someone voted for a game - in effect its binary, either someone voted for it or they didn't, thereby not taking into account how much they liked it.

Well steam also provides only a binary assessment of quality, you are forced to choose between making a positive or negative stance.

So if someone wanted to express the fact that they thought XYZ game was 'mediocre' via a Steam review, would they choose to make a 'positive' review or a 'negative' review? And, yes, I would say this is, again, more relevant to any discussions about PoE and it's 'score'.

Nereida said:

Meanwhile, I made this other statistic pane. It is valid because I didn't vote in Steam, and so it shows a perfectly scientifically accurate value.

Since I have a high opinion of myself and I arbitrarily decided that me and my buddies (but mostly me) are the ones who knows most about RPGs and hold the absolute rule on what is the quality of them, there is no better place to look at than averaging my own opinion in a 1:1 proportion against the opinions of thousands, for an accurate statistical value.

Click to expand...

You don't understand how polling works, that's fine.

Most Gallop polls only poll a thousand or so people, and they are usually quite accurate the wider population. And used by professionals.

But feel free to continue being as disingenuous as you possibly can be.

And I'm glad you now realise population is in fact a relevant factor. I look forward to you apologising for saying it was irrelevant earlier and admitting your were wrong.

You are entitled to your opinion, but I already stated that I am done educating you. It is not my responsibility.

WotR was a pretty good game, if not the best I, or anyone I can tell has played. But to prove my fondness for it, I will borrow a trait from one of its characters, Nenio.

"What is lackblogger? It was of no relevance, and had no value to me, so I chose to forget it".

I would rather you enlightened me on how one makes a 'mediocre' steam review

I would rather you explained why population suddenly matters to you so much when, earlier in the thread, it had zero relevance.

Nereida said:

me and my buddies (but mostly me) are the ones who knows most about RPGs

Click to expand...

I thought that was Felipe ?

lackblogger said:

I would rather you explained why population suddenly matters to you so much when, earlier in the thread, it had zero relevance.

Click to expand...

There are 2 approaches to it.

1. What we call "herd / swarm intelligence". It assumes that the "big thing" is like = all individual intelligences added together to 1 meta intelligence.

2. The other one is like the saying : "Millions of flies cannot lie".

OK, it took me a bit longer to finish it, I had some other things to do, but here's the link to the spreadsheet. And the charts are once again attached. Rules of the vote can be found here.

That's an awesome set of stats.

However, remember that's the codex, they are not a general RPG website that welcomes any and all opinions. Their top 2 are extremely sacred holy cows and any post, no matter how vague or tame in its criticism of those games will be responded to thusly:



So it's no surprise those top two stay at the top, you might as well be posting on a Fallout and PS:T dedicated forum. If those two games ever stop being the no.1 and 2 spot the site would likely self-combust and cease to exist.

So I think it would be of better statistical value to remove those 2 games as outliers by default, because there's no generalised opinion on them allowed there.

BG2 is, obviously, the all-time outlier equivalent to Witcher 3 on our list.

So, just for interest, if not for very valid reasons, it would be nice to see the chart with those 3 removed as outliers, which they are anyway. BG2 in particular fulfils all of the criteria discussed anyway, as did Witcher 3, that of both high scores AND high population. The undeniables.

Also I notice that for 41 games you have written N/A for critic score and 44 games are N/A for user score. How have you charted these? Especially as 6 out of the top 30 are N/A.

Did you not want to include them as the Critic and User score be zero? Surely if there's no historical record available on-line score for a game and it still resonates over time, isn't that quite heavy proof that scores aren't so important? But then they probably would have had scores and hype in magazines once upon a time, so maybe they did have scores once. But then since no-one can see those scores now are they relevant anyway, etc.

Games without score were discarded from the chart. I don't know if that was a good way to do it because they were still voted on and that might possibly affect the points of other games. The point was to get a larger data set than our vote. I know the Codex is quite biased but same can be said about our vote as well, or any niche community for that matter. To get unbiased results, I would have to use data from a global vote.

I removed the games you mentioned but that requirement is arbitrary. For example, I noticed a lot of games which I personally wouldn't consider "RPG" getting low amount of votes despite having high scores. It is possible that a lot of people agree with me and chose not to vote for them, even though they might think it's a good game. I chose to leave them there because my point is to use the data I have. I think it's not really a good idea to do this kind of analysis with specific genres, especially RPG since 100 people will give you 200 opinions on what is and what isn't an RPG. I think a general set of games, regardless of the genre would have been better. But again, I did this as a fun exercise and this is the data I have. I still believe it can tell us something, but it is not perfect.

All I know is that if an IGN or a Gamespot gives a game a 10/10 I'm immediately skeptical about it given game journos renowned preference for easy, little difficulty games. I value gameplay highly so adverts showing these sorts of scores kinda highlight that I should regard it with a wary eye.

Then I will assess what players with similar tastes to myself think about the game and I find steam reviews to be a great help here as I am quickly able to discern the type of player they are by the content of the review. As far as legacy goes favourites get re-assessed all the time. At one time Kotor was regarded as the better and then at another time Kotor 2. Changing trends and fashions empathise different qualities and things that weren't trendy at the time become so.

What we don't have at present time are official genre experts who go over classic games and assess them for their inherent qualities for posterity as in other media. I actually think this is a good thing as I find mainstream sources dubious at best. YouTubers have filled this gap to a large degree because they go over the games in depth to an extent that games media rarely does. As such I believe legacy will be determined by influential Twitch and YouTube personalities until legacy media tries to kneecap them.

P.S All the 'best' authors and artists are regarded more favourably upon their deaths than when alive.

I tend to find that average user score is more meaningful for me than average critic score. Too many critics these days give politics etc in games too much score now.

I find that even though that user score is prone to review bombing, there isnt enough review bombing to impact the score that much in user scores.

Occasionally fans of the previous game review bomb the sequel because of how the game disrespects the original like Last of Us 2. But those are rare.

What I found during the analysis is that user and critic scores usually match, however sometimes they don't. Sometimes it can be a smaller number of, usually critic reviews, that affects the final score. For example, if you only have three critic reviews with scores of 85, 81 and 53, the average would be 73. The number of user reviews could be greater and average user score could be 85 and the difference between 85 and 73 is quite noticeable. However, the 53 score could be an outlier and is lowering the average score to 73. If we remove it, the average critic score rises to 83 which is much more in line with the user score.

I personally like to read/watch the entire review to get an idea about the game. I also watch gameplay videos/streams to get an idea of how the game plays like. It is also beneficial to know the reviewer's preferences. For example, if a reviewer says that a game is too grindy and we know that they really, really don't like grindy games the game might not actually be too grindy for me. Maybe the reviewer has zero tolerance for such things but I have some tolerance or even enjoy it. Reviewer might end up giving the game a lower score than I would do because of that, or his con point might be my pro point. Or if a reviewer mentions that the game doesn't have a manual save function and they are not bothered by that, it may not lower the score. However, for me that could be a huge minus, even to the point of not buying the game (even though the rest of the game might be to my liking).

I remember 15 years ago all I had to go on were magazine reviews and sometimes I would get interested on the game based on something the reviewer said or by a screenshot and when I eventually did get the game it turned out to be nothing like I imagined. Screenshots tell us 1/1000 of what gameplay videos tell us, and gameplay videos tell us 1/1000 of what actually playing the game is like. The amount of info you can gather before you decide if you want to get it is much higher now than it was 15 years ago, but still you will know whether or not you like the game only after you play it.

I really like PS:T, so any statement or stats claiming the game is not awesome is obviously flawed and therefor false

Myrthos said:

I really like PS:T, so any statement or stats claiming the game is not awesome is obviously flawed and therefor false

Click to expand...

Like or liked?

I've refenced my back yard a few times, but I've never refenced quality. I didn't even know it needed a fence in the first place. I am the only one to directly tackle the original topic. I really don't see the need to refence anything scores of times to indicate the quality of a good fence job.

Skyrim rules!

Ivanwah said:

Games without score were discarded from the chart. I don't know if that was a good way to do it because they were still voted on and that might possibly affect the points of other games. The point was to get a larger data set than our vote. I know the Codex is quite biased but same can be said about our vote as well, or any niche community for that matter. To get unbiased results, I would have to use data from a global vote.

Click to expand...

The codex isn't 'quite biased', it's fanatical to the point extreme, it's the kind of site that actively encourages the concept of 'site consensus' as a means to validate itself and it's reasons to tar and feather people.

You say you wanted a larger data set, but the irony is that your primary source is actually a tiny data set anyway.

Take Lords of Xulima, for example. It's Critic score is based on just 9 individuals giving their opinion from a general gaming perspective. One of them out right says "Veteran players will be pleased, the others not so much." - well who's going to be interested in it if not veteran RPG players, who is the reviewer even speaking for when they say that?

Another writes "Lords of Xulima is strictly for the more masochistic of gamers." Which is absurd. Anyone who's even the slightest bit into RPGs will be able to pass through the game just fine, again, who are they even speaking to? Clearly not RPG specialists.

So these 9 people decide the 'longevity' score for something that isn't even their main bag? LOL, don't be absurd. LoX a 71? Absurd.

That data set is SIGNIFICANTLY smaller than the RPGWatch dataset.

The game has 33 User reviews - that's not a 'larger sample size'. As a result just 3 people who gave it a negative review have dumped it's user score down to 7.7. Well, that was extremely easy to bully out the way wasn't it.

Even combining the two, that's just 42 individuals providing your primary reference stat. And in that 42 you have people who don't even care for RPG specialisation.

The RPGWatch poll is actually a larger, and more relevant, sample size than the stat that Nereida is holding up as the DEFACTO reference to a game's UNDISPUTED quality.

So lets now look at steam reviews: 1,110 reviews, very positive, the same as Dov Os and PoE. This isn't to say "someone who likes PoE will like LoX and visa-versa, just that they are likely of similar quality to those who specialize in RPGs.

The RPGWatch 'sample size' was just fine. And provides a more relevant assessment of 'quality'.

And from the larger sample size you took from the codex, your red line of Critic Score versus Codex Score still shows a very similar arc to the RPGwatch one, that, yes, Score has a small effect on the likelihood of a game having a longevity of favour.

Their Site-Score versus User-Score is indeed more marked. And I must admit, I usually only look at user scores on Metacritic, due to the 'official reviewers' usually being just random individuals with no specific specialisation and who knows what axes to grind.

crpgnut said:

I've refenced my back yard a few times, but I've never refenced quality. I didn't even know it needed a fence in the first place. I am the only one to directly tackle the original topic. I really don't see the need to refence anything scores of times to indicate the quality of a good fence job.

Skyrim rules!

Click to expand...

Just the person we're looking for atm!

Your opinion is required in the following thread about a survival game telling us it's RPG:

https://www.rpgwatch.com/forums/showthread.php?t=48557

By larger data set I meant more people voting on more games. If I remember numbers correctly 361 people voted in their vote for 279 games. In our voting thread there were 311 posts but a lot of them were not people voting, and 50 games ended up getting a vote. 361 > 311 (I didn't count how many were votes so this number should really be lower) and 279 > 50.

Edit: scratch that, the votes were sent by PM to you so you can tell us how many people actually voted. Maybe that information is available somewhere, but I missed it.