Metacritic Matters: How Review Scores Hurt Video Games

[Brother None]

Kotaku has a pretty interesting piece up about how the games industry uses Metacritic, and why it's not a good thing. This New Vegas example shouldn't be news to people though the amount cited is.<blockquote>Perhaps you've heard the story: publisher Bethesda was due to give developer Obsidian a bonus if their post-apocalyptic RPG averaged an 85 on Metacritic, the review aggregation site. It got an 84 on PC and Xbox 360, and an 82 on PS3.

“If only it was a stable product and didn't ship with so many bugs, I would've given New Vegas a higher score,” wrote a reviewer for the website 1up, which gave New Vegas a B, or 75 on Metacritic's scale.

“It's disappointing to see such an otherwise brilliant and polished game suffer from years-old bugs, and unfortunately our review score for the game has to reflect that,” said The Escapist's review, which gave the game an 80.

If New Vegas had hit an 85, Obsidian would have gotten their bonus. And according to one person familiar with the situation who asked not to be named while speaking to Kotaku, that bonus was worth $1 million. For a team of 70 or so, that averages out to around $14,000 a person. Enough for a cheap car. Maybe a few mortgage payments.</blockquote>

 

[AtomBomb]

Wow...I never realized how little these people make off of making a video game. So does most of the money made go to the publisher?

 

[Faceless Stranger]

“If only it was a stable product and didn't ship with so many bugs, I would've given New Vegas a higher score,” wrote a reviewer for the website 1up, which gave New Vegas a B, or 75 on Metacritic's scale.

Hmm, funny, bugs didn't stop them from giving Fallout 3 an "A"

 

[Gaspard]

Hmm, funny, bugs didn't stop them from giving Fallout 3 an "A"

word

 

[Bewitched]

Brian Fargo would do Wasteland 2 for 1 million bucks.

 

[woo1108]

Hmm, funny, bugs didn't stop them from giving Fallout 3 an "A"

word

scoring 100 dispite of bug, shitty system, poor writing and voice acting, disconnection between 1 and 2 etc.
Good point? maybe visual? I don't like fo3's visual though.
And dispite of bug is beth's fault, lots of idiots blames Obsidian.

 

[Crni Vuk]

“If only it was a stable product and didn't ship with so many bugs, I would've given New Vegas a higher score,” wrote a reviewer for the website 1up, which gave New Vegas a B, or 75 on Metacritic's scale.

Hmm, funny, bugs didn't stop them from giving Fallout 3 an "A"

Marketing. Its all about the Marketing with games today. Reviews like Metacritic dont really review the "game" but actually the marketing campaign by the Publisher/Developer. There are exceptions of course - See Diablo 3. It had to face a lot of criticism.

But usually the better the marketing and hype, the higher the scores.

Wow...I never realized how little these people make off of making a video game. So does most of the money made go to the publisher?

not only.

A good thing to look for is music, because a lot of things are similar. I dont have all the details in head, but since many games still are released with an box and DVD a large part of the money goes well, to the production.

 

[Beelzebud]

I think the thing that annoys me about the situation with Obsidian, is that everyone seems to give Bethesda a complete pass on the exact same issues. I found Fallout 3 to be more buggy than New Vegas, and yet everyone that reviews games acts like it was perfect.

 

[Walpknut]

Fallout 3 was way more buggy than New Vegas, and New Vegas had more content and less development time, ain't that funny?

 

[Stanislao Moulinsky]

“A lot of times when we're talking to publishers–and this is no specific publisher–but there are conversations I’ve had in which the royalty that we could get was based upon getting a 95,” he said. “I’ve had this conversation with a publisher, and I explained to them, I said, ‘Okay, there are six games in the past five years who have averaged a 95, and all of those have a budget of at least three times what you’re offering me.’ They were like, ‘Well, we just don’t think we should do it if you don't hit a 95.’”

One reviewer well-known for aberrant scores is Tom Chick, who runs the blog Quarter To Three. Chick is listed for having the lowest Metacritic score on BioShock Infinite (a 60) and Halo 4 (a 20), among others. He uses a 1-5 scale that Metacritic converts into multiples of 20, so Chick’s “I liked this game,”–3 out of 5–is converted into a 60, which most Metacritic readers see as a bad score.

One of Doyle’s other big policies has also been in the news recently: Metacritic’s refusal to change an outlet’s first review score, no matter what happens.[...]Sometimes, however, this leads to some skewed Metacritic results. Late last year, GameSpot pulled their review of Natural Selection 2, which had been written by a freelancer. The review contained multiple factual inaccuracies. A different writer then reviewed the game, giving it an 8. But the original score–a 60–remains on Metacritic to this day.

More recently, the website Polygon, which uses an adjustable review scale, gave SimCity a 9.5 out of 10 before it launched. On launch day, when crippling server errors rendered the game unplayable for most, Polygon changed their score to an 8. A few days later, as the catastrophic problems continued, they switched it to a 4. It's currently a 6.5. Yet anyone who goes to SimCity's Metacritic page will still see the 9.5.

The Metascore is a flawed concept based on a flawed system used in a flawed way. It boggles my mind that people genuinely think that it's a good metric or see no problem in the way the industry uses it.

 

[Kwiatmen666]

The Metacritic's Metascore indeed evaluates mainly the marketing and hype around the game - the examples of this are all three of latest games developed by Bethesda: Oblivion, Skyrim and Fallout 3. Their writing, and storytelling are disappointingly mediocre and such a thing as "depth" as we know it from other rpgs doesn't seem to exist for Beth's lead designers. I personally do not even think those games should be called rpgs - these are just some kind of open-world-based action-adventure games with elements of rpg. And - arguably - dialogs. Kind of... But don't take me wrong - I think Skyrim is quite a game in its own measure - the design of the world itself is great, but what's in it is entirely different matter.

As I played all of Bethesda's latest games, I played all of Obsidian's games - Kotor II, NWN 2, Alpha Protocol, New Vegas - and let me tell those of you who didn't - the difference in writing, storytelling, character design are all COLOSSAL. Such ambition and passion with which Obsidian makes their games echoes in every aspect of their products. This can't be said about Bethesda. But the passion and ambition are also traps - in every of their games Obsidian leaves quite visible traces of unfinished quests and plot-lines. And this also can't be said about Bethesda - they seem to plan carefully and take their time to wrap everything up while Obsidian wants to add more than humanly possible (not a bad thing imo...) and so while striving for perfection, it only gets further away.

A sad thing really - cold calculation and devious advertising campaign yealds better "numbers" than true passion and dedication I see in every game of Obsidian's.

I was thinking about something constructive, but that long-winded rant is all I could come up with. Sorry;p

 

[Stevie D]

As I played all of Bethesda's latest games, I played all of Obsidian's games - Kotor II, NWN 2, Alpha Protocol, New Vegas - and let me tell those of you who didn't - the difference in writing, storytelling, character design are all COLOSSAL. Such ambition and passion with which Obsidian makes their games echoes in every aspect of their products. This can't be said about Bethesda. But the passion and ambition are also traps - in every of their games Obsidian leaves quite visible traces of unfinished quests and plot-lines. And this also can't be said about Bethesda - they seem to plan carefully and take their time to wrap everything up while Obsidian wants to add more than humanly possible (not a bad thing imo...) and so while striving for perfection, it only gets further away.

A sad thing really - cold calculation and devious advertising campaign yealds better "numbers" than true passion and dedication I see in every game of Obsidian's.

I was thinking about something constructive, but that long-winded rant is all I could come up with. Sorry;p

No, I think that was well said. And it probably explains why I finished FO3 and enjoyed it more than F:NV, which I let slip as the NCR vs. Caeser's Legion thing started coming to a messy head.

I guess I prefer a mediocre idea well executed to a good idea sloppily executed.

That said, the graphics engine used for the two had outstayed its welcome by the end of FO3, so F:NV had that particular hand tied behind its back.

 

[woo1108]

The Metacritic's Metascore indeed evaluates mainly the marketing and hype around the game - the examples of this are all three of latest games developed by Bethesda: Oblivion, Skyrim and Fallout 3. Their writing, and storytelling are disappointingly mediocre and such a thing as "depth" as we know it from other rpgs doesn't seem to exist for Beth's lead designers. I personally do not even think those games should be called rpgs - these are just some kind of open-world-based action-adventure games with elements of rpg. And - arguably - dialogs. Kind of... But don't take me wrong - I think Skyrim is quite a game in its own measure - the design of the world itself is great, but what's in it is entirely different matter.

As I played all of Bethesda's latest games, I played all of Obsidian's games - Kotor II, NWN 2, Alpha Protocol, New Vegas - and let me tell those of you who didn't - the difference in writing, storytelling, character design are all COLOSSAL. Such ambition and passion with which Obsidian makes their games echoes in every aspect of their products. This can't be said about Bethesda. But the passion and ambition are also traps - in every of their games Obsidian leaves quite visible traces of unfinished quests and plot-lines. And this also can't be said about Bethesda - they seem to plan carefully and take their time to wrap everything up while Obsidian wants to add more than humanly possible (not a bad thing imo...) and so while striving for perfection, it only gets further away.

A sad thing really - cold calculation and devious advertising campaign yealds better "numbers" than true passion and dedication I see in every game of Obsidian's.

I was thinking about something constructive, but that long-winded rant is all I could come up with. Sorry;p

I don't think beth's games are well-made for it's genre.
They always makes game buggy, unbalanced system that has critical flaw(Daggerfall and Oblivion's level scailing is just sucks)(Fo3's Vats system, location design, skill, SPECIAL, journal which from TES4, etc). For skyrim, one of the TES's good point was faction quest. But that faction quest is ruined and fo other side quests, they use TES2's way that was just copying other quest and only different thing is what NPC saying and that saying is poor written.

Compare with beth's flawed game, I rather choose Obsidian.

I don't think metacritic is fair. For FO3 that has lots of flaws that other game had criticised and gain infamy for that but they got high score. and I don't think fo3 has that good point to cover bad points to gain that.

 

[Kwiatmen666]

No, I think that was well said. And it probably explains why I finished FO3 and enjoyed it more than F:NV, which I let slip as the NCR vs. Caeser's Legion thing started coming to a messy head.

I guess I prefer a mediocre idea well executed to a good idea sloppily executed.

Yeah, I bet everyone has a little different take on this. I hated Fallout 3 for its shallowness and so I was a bit anxious to pick up New Vegas, but I was very pleasantly surprised.

That said, the graphics engine used for the two had outstayed its welcome by the end of FO3, so F:NV had that particular hand tied behind its back.

That goes without saying, but to be honest - I don't think Obsidian's games have ever exceeded in graphics. On the other hand rpg as a genre was never on the top of the cutting-edge graphics development, and so I don't think we - rpg fans - should give too much thought. In fact, focusing a review of an rpg on graphics is like trying to judge cookie's taste on the basis of its shape. Rather ridiculous, right?;p

 

[Ares]

TB has been raging about Metacritic as well for as long as I can remember.

But Metacritic isn't going anywhere anytime soon so I would like to ask, how to at least control the damage?

For one I wonder if imposing and enforcing a unified "rating" system would help (yes, yes: I know that it in itself is a dumb system to judge something like entertainment but lets be honest: it also is not going anywhere anytime soon)- for one thing it would eliminate all the supidness like converting a 5 point system into a 100 scale, which seems to cause all manner of statistical inaccuracies in the metacritic weighted average equation.

The best place IMO would probably be by Metacritic itself- for example a contract requirement would be that in order to have your reviews represented in the metacritic equation you need to adopt our parameters. As far as I have learned anything about economical statistics and analyzsis one of the prerequisites is that the units that your data is based on has to be converted into a unified scale first.

 

[Arden]

Nah, the moment you try to correct at least some things in a fundamentally flawed system you only add to its lifespan and legitimacy. Metacritic is a fundamentally flawed system used to make fundamentally false decisions. Even marketing guys know this but there are no other "numbers" to base any decisions upon for now. Sales tell you how many bought the game, but not if they enjoyed it.

 

[Stevie D]

That goes without saying, but to be honest - I don't think Obsidian's games have ever exceeded in graphics. On the other hand rpg as a genre was never on the top of the cutting-edge graphics development, and so I don't think we - rpg fans - should give too much thought. In fact, focusing a review of an rpg on graphics is like trying to judge cookie's taste on the basis of its shape. Rather ridiculous, right?;p

To a degree, although Fallout 1 and 2's audio and visual aesthetics were a big part of the games' draw. Pretty eye- and ear-popping stuff back in the day

 

[gumbarrel]

But usually the better the marketing and hype, the higher the scores.

So all games like Fallout 1 and Arcanum need to do is market the fuck out of themselves and they will sell trillions, right?

 

[warsaw]

So all games like Fallout 1 and Arcanum need to do is market the fuck out of themselves and they will sell trillions, right?

Hyperbole aside... uh, yeah. You market a game right, and people will buy it.

 

[gumbarrel]

So all games like Fallout 1 and Arcanum need to do is market the fuck out of themselves and they will sell trillions, right?

Hyperbole aside... uh, yeah. You market a game right, and people will buy it.

And yet that's not what anyone here ever actually argues. I've never seen anyone say "Man, Title X should get the highest possible marketing budget, so it can sell boatloads!" If you really think that marketing is the end-all of game sells, then you should want companies like Obsidian to increase their marketing budget, but that's not something that anyone actually argues, ever. The opposite, actually.

Not to mention the fact, that marketing isn't brain-control magic and you have plenty of very well marketed games that flop big time. Dante's Inferno, Lair, Brutal Legend, etc.