Steam Reviews

[Lorini]

Today on Steam they announced a change to Steam Reviews. Thoughts? I know in some of the threads we've talked about the (un)reliability of the reviews, this may help. I also (and them too) realize that the key filter defaulting to off will result in reviews by GMG buyers being ignored, but perhaps that's a sacrifice that has to happen. Also since Steam started issuing refunds, the 20% less you get the game from GMG may be offset by the chance to get a refund so I wonder how many people are still using GMG regularly.

Steam keys have always been free for developers to give out or sell through other online or retail stores. That isn't changing. However, it is too easy for these keys to end up being used in ways that artificially inflate review scores.

An analysis of games across Steam shows that at least 160 titles have a substantially greater percentage of positive reviews by users that activated the product with a cd key, compared to customers that purchased the game directly on Steam. There are, of course, legitimate reasons why this could be true for a game: Some games have strong audiences off Steam, and some games have passionate early adopters or Kickstarter backers that are much more invested in the game.

But in many cases, the abuse is clear and obvious, such as duplicated and/or generated reviews in large batches, or reviews from accounts linked to the developer. In those cases, we've now taken action by banning the false reviews and will be ending business relationships with developers that continue violating our rules.

While helpful users in the community have been valuable in reporting instances of abuse, it's becoming increasingly difficult to detect when this is happening, which reviews from Steam Keys are legitimate, and which are artificially influenced.

This probably also has to do with the difficulty of trying to start a game off of Steam, promising Steam keys when the game is released, and then giving those keys out. Apparently devs can no longer do that.

I use the reviews there sparingly. I didn't realize (how could I) that so many of the positive reviews were generated by bogus dev keys. I am concerned about devs now basically being forced into early access for their alpha/beta because asking people to buy the game from their website with no promise of a Steam key is going to cut down that activity by quite a bit. As someone who bought both Rimworld and Factorio from the website, it'd be sad to have to buy the game again to get it on Steam.

[Defiant]

An analysis of games across Steam shows that at least 160 titles have a substantially greater percentage of positive reviews by users that activated the product with a cd key, compared to customers that purchased the game directly on Steam.

Aren't there something like 8000+ games on steam? So we're talking about 2%?

This probably also has to do with the difficulty of trying to start a game off of Steam, promising Steam keys when the game is released, and then giving those keys out. Apparently devs can no longer do that.

I'm confused. Does that mean if you buy a game from a developer outside of steam, and they want to add the game to steam, they won't be able to give out steam keys to the users who bought it previously? That would suck. But I'm not sure why that would contribute to the problem if those people legitimately bought the game.

Will this effect things like humble bundle and similar bundles?'

Edit: OK, so it sounds like nothing will change, they'll still be able to, except that their reviews won't count towards the score. Which is a shame when it's legitimate buyers outside of steam, but not as severe a change as it sounded it might be.

[tgb]

If a developer wants to skew the reviews by giving keys away to friends, he or she is going to do so anyway, even if it means reimbursing the friend for purchases directly from Valve. I don't see this affecting the average user one way or another.

As to whether people still buy from GMG or CDKeys or whoever knowing that the discount will mean no refund, I still do it - but I do enough homework to make sure I'll be happy with the game first.

[Defiant]

If games are bought from Steam and then refunded, will the reviews count or not count under this new system?

[Rumpy]

Yeah, I don't pay attention much to the reviews either. But I do wonder how this will affect early-access in general, not just on Steam but on other sites as well. I see it as a good thing as well, though I'm sure there will be growing pains until the dust settles. I like the way Amazon does it by giving the reviewer a "Purchase Verified" statement beside the reviewer's name (which I'd actually prefer over this), but at the same time, they don't discourage outside purchasers from making reviews and you can usually get a feel for those that are genuine, and they can be useful. But part of it is common sense, I think. I wouldn't be writing a review of a Steam purchased game on GOG or vice-versa.

[Blackhawk]

This is necessary. Steam reviews had reached a point at which they had become irrelevant. They said they knew of at least 160 games that have used this exploit, but that they're still researching it. It is probably more. They also talk about some of the manipulation and abuse of the system by small groups of users (the GamerGate bullshit where groups would systematically attack games with any feminist content by boosting all the negative reviews and voting down all the positive ones is a good example.)

The only changes they're making are that they're adding a filtering system to the reviews, and that games acquired from keys rather than directly through Steam would not have their reviews included in the aggregate positive/negative total at the top. They are also going to crotch-kick the developers who have been engaging in the abuse.

As far as developers giving out keys, the only thing I'm aware of was when, almost a year ago, Valve told Greenlight developers to stop giving out free keys in exchange for votes to get the game made. It was breaking the system. They can still give keys away for other reasons, or sell them, or write the on slips of paper and toss them out of a plane.

[Blackhawk]

And to clarify, you can still review any game you own on Steam, regardless of where you got the key. It just won't count toward the overall positive/negative percentage.

[Lorini]

If a developer wants to skew the reviews by giving keys away to friends, he or she is going to do so anyway, even if it means reimbursing the friend for purchases directly from Valve. I don't see this affecting the average user one way or another.

As to whether people still buy from GMG or CDKeys or whoever knowing that the discount will mean no refund, I still do it - but I do enough homework to make sure I'll be happy with the game first.

Um, there's a big difference between being able to give out a few dozen keys for free and buying them. Many indie games are above $15. Even for just 10 games, that'd be $150. I don't think they'll be doing much of that. As far as percentages go, while this may only affect 2% of the games, that's probably about the percentage of games that people care about the reviews for.

[Defiant]

I do look at reviews of games, including the overall score. But more if the overall score is bad (that way I can avoid the rare game that is absolutely horrible)

[Nightwish]

If they're not doing anything about one line reviews that say nothing whatsoever, they clearly don't care about the quality of their system.

[Smoove_B]

Yeah, it's one of those things that I don't know how they fix. Clearly its sometimes useful, but they're absolutely filled with nonsense reviews (good or bad). I use it for some information but I don't think it would be the sole arbiter in determining a purchase.

[MindToyGames]

I've spoken on Qt3 about what I think of this 'change'. I think it's heavy-handed and punishes those who rely on word of mouth and their fanbase to spread the word about their games. Now many small indies have to rely on random Steam users, who, let's face it, aren't very into niche type strategy games. I think it'll cause a big ripple effect for indies who sell niche games on Steam, and not a positive one. At least 6 other devs i've spoken to feel the same. Hopefully they fine-tune this policy as time goes along.

[tgb]

I look at reviews on Steam the same way as I do Yelp or Tripadvisor - I assume the highest 20% are friends and family and the lowest have an axe to grind, and just pay attention to the ones in the middle.

The only exceptions are items with reviews that lean one way or another by a huge margin.

[Lorini]

I've spoken on Qt3 about what I think of this 'change'. I think it's heavy-handed and punishes those who rely on word of mouth and their fanbase to spread the word about their games. Now many small indies have to rely on random Steam users, who, let's face it, aren't very into niche type strategy games. I think it'll cause a big ripple effect for indies who sell niche games on Steam, and not a positive one. At least 6 other devs i've spoken to feel the same. Hopefully they fine-tune this policy as time goes along.

I don't understand this at all. From here on out, the default view will not include reviews where the reviewer didn't pay for the game. That's it. Where do you draw all of this other stuff from? Of course the devs don't like it, they can't skew the reviews anymore unless they pay for keys to give away. Word of mouth will still get people who are considering buying the game to buy the game. If there's a real fan base, then that fan base will still there. Creating a fake fan base is bullshit and unethical. As a consumer I'm happy for the change.

[Blackhawk]

I think that the argument against it has more to do with the fact that this affects a broad swathe of the industry while the abusers represent a tiny fraction of that. I have been a reviewer, and I have been a reviews manager (who coordinated between the site, the reviewers, and the publishers), and it is perfectly normal business, from the AAA publishers on down, to give out games to reviewers. It is how you get reviews on the books.

The issue is the tiny number of devs who manipulated that into only positive reviews (bot reviews, paid review companies, etc), but it also blocks all those who send out dozens of keys to established reviewers in order to get the ball rolling, who count on the quality of their products to bring about good reviews.

The real problem is that Steam reviews are way, way over the credibility cliff already. I'm not sure what Steam could do short of nuking every review on the books and restarting from scratch.

[Lorini]

So what if they presented a ratio? Let's say for every x number of non key reviews, there can be a key review posted. Would that help? Flooding the game with reviews from key owners is doing a disservice to consumers but if they were limited/controlled then that would be OK.

In other review news Valve has banned an indy dev for suing players who gave it bad reviews. One would hope not only do these guys lose the suit but that they are countersued into oblivion. Reviews would become worthless if every time a bad review was made, the person would be subject to losing a lawsuit, in fact even the specter of being sued and having to defend against a lawsuit would turn away a lot of people from publicly offering a negative opinion.

[Blackhawk]

So what if they presented a ratio? Let's say for every x number of non key reviews, there can be a key review posted. Would that help? Flooding the game with reviews from key owners is doing a disservice to consumers but if they were limited/controlled then that would be OK.

I think that would be a morass to moderate. I think a 'certified reviewer' system would be easier.

On the whole, Steam has started to get an unsavory reputation, at least as far as the community and indie scene goes (Steam reviews, Greenlight abuse, shovelware, GamersGate abuse, that sort of thing.) It sounds like they've decided to start cleaning that up. Which is good. We need to be able to trust Steam.

[Lorini]

So what if they presented a ratio? Let's say for every x number of non key reviews, there can be a key review posted. Would that help? Flooding the game with reviews from key owners is doing a disservice to consumers but if they were limited/controlled then that would be OK.

I think that would be a morass to moderate. I think a 'certified reviewer' system would be easier.

On the whole, Steam has started to get an unsavory reputation, at least as far as the community and indie scene goes (Steam reviews, Greenlight abuse, shovelware, GamersGate abuse, that sort of thing.) It sounds like they've decided to start cleaning that up. Which is good. We need to be able to trust Steam.

OK I see what you mean. What I was thinking of was to take a random key generated review and insert it for every x number of verified purchase reviews. Yes Steam has started to get an unsavory reputation and I guess that's why Facebook thinks it can compete in this business. I can't see that happening but clearly Valve is taking steps to be more consumer friendly.

[tgb]

I think that the argument against it has more to do with the fact that this affects a broad swathe of the industry while the abusers represent a tiny fraction of that. I have been a reviewer, and I have been a reviews manager (who coordinated between the site, the reviewers, and the publishers), and it is perfectly normal business, from the AAA publishers on down, to give out games to reviewers. It is how you get reviews on the books.

True, but aren't you talking about giving games to the professional reviewers like Tom Chick or Troy Goodfellow, or someone from IGN? People like that don't tend to post Steam reviews anyway.

I've yet to read a review on Steam that didn't read like it was written by someone (at best) in high school.

[MindToyGames]

So what if they presented a ratio? Let's say for every x number of non key reviews, there can be a key review posted. Would that help? Flooding the game with reviews from key owners is doing a disservice to consumers but if they were limited/controlled then that would be OK.

I think that would be a morass to moderate. I think a 'certified reviewer' system would be easier.

On the whole, Steam has started to get an unsavory reputation, at least as far as the community and indie scene goes (Steam reviews, Greenlight abuse, shovelware, GamersGate abuse, that sort of thing.) It sounds like they've decided to start cleaning that up. Which is good. We need to be able to trust Steam.

OK I see what you mean. What I was thinking of was to take a random key generated review and insert it for every x number of verified purchase reviews. Yes Steam has started to get an unsavory reputation and I guess that's why Facebook thinks it can compete in this business. I can't see that happening but clearly Valve is taking steps to be more consumer friendly.

I could probably live with that. In my case, I have a genuine pool of people who are fans of my niche games, and are not coerced in ANY way to leave a positive review. They choose to, which should matter. Now those reviews get voided as far as default Store page visibility, which is neither fair nor right. It will make it (IMHO) an uphill battle for niche games to get noticed on Steam, because you're depending on random floating Steam users, who tend as a general rule not to be the more mature, thoughtful reviewers in general who will give a niche strategy game a shot past around the 10 minute mark or so.

My recent game has 10+ legitimate, well-thought out reviews that now count for nothing on the main Store page score, which is frustrating beyond words.

[Lorini]

If they didn't pay for the game, they have nothing invested in the game, and can freely leave a positive review even if they didn't like it. They may likely think that if they leave a negative review they won't get more free games, so why would they do that? By giving a positive review for something they got for free they appear to have a conflict of interest, even if they actually don't. And as someone who was involved with ethics for over 10 years, the appearance of a conflict of interest does as much damage as an actual conflict of interest.

I'm sorry you think that giving away keys and then having those people leave reviews is a way to build momentum for your brand. The US FTC has been cracking down on this practice for years, it's time Valve got with the program.

[Zarathud]

A 99% off promotional offer to the same pool accomplishes the same goal, and technically complies with the rule.

[Rumpy]

I think doing it like Amazon and putting a "Steam Verified Purchase" badge next to a review might be an attractive solution, without having to single out or separate any reviews. Also, instead of punishing those bought using an outside source, I wonder if it might be possible for Steam to acknowledge them so that if say, a game were bought via a Humble Bundle, instead of it having a Steam identifyer, then having something that says "Bought via HumbleBundle" or something to that effect and any keys that can't be identified wouldn't get any idenitifyer. It might even encourage higher quality reviews.

[tgb]

A 99% off promotional offer to the same pool accomplishes the same goal, and technically complies with the rule.

So does having a bunch of reviewers pay for the game on Steam and then reimbursing them. There are all kinds of ways to get around the new rules.

[Lorini]

A 99% off promotional offer to the same pool accomplishes the same goal, and technically complies with the rule.

So does having a bunch of reviewers pay for the game on Steam and then reimbursing them. There are all kinds of ways to get around the new rules.

I think Valve (and the FTC) want to make it harder. So sure they can have them pay and and then reimburse, but again like I said before, that's money out of their pocket. That's actually fine by the way, if they want to pay and gift reviewers at least the game was paid for by someone. They are filtering out the gift keys which is the same as 'verified purchase', but unlike on Amazon, you can only get Steam games on Steam, so the filtering actually means something.

[Blackhawk]

I think that the argument against it has more to do with the fact that this affects a broad swathe of the industry while the abusers represent a tiny fraction of that. I have been a reviewer, and I have been a reviews manager (who coordinated between the site, the reviewers, and the publishers), and it is perfectly normal business, from the AAA publishers on down, to give out games to reviewers. It is how you get reviews on the books.

True, but aren't you talking about giving games to the professional reviewers like Tom Chick or Troy Goodfellow, or someone from IGN? People like that don't tend to post Steam reviews anyway.

Yes, but that system has alwsys been used for the smaller sites as well. They may not get as many games, or as good of exclusives, but they still get product to advertise. There are a number of reviewers on Steam who do write professional quality reviews who are subsequently noticed by developers and offered free keys in exchange for honest reviews (just like Tom Chick or Troy Goodfellow.) This sadly neutralizes them along with the dishonest ones.

I still think a 'certified reviewer' status that guarantees both visibility and inclusion regardless of key source would allow this system to work without penalizing the best reviewers on Steam, or the developers that work with them.

[dbt1949]

When I read the reviews I find my attention span wandering after a paragraph or two. Many are just too damn long. And people who write a review after playing less than an hour are more useless than ticks on a hound.

[TheMix]

Lorini, you seem to be assuming that the keys were provided for free. My understanding (and I could be completely wrong) is that in some/many cases the keys are not sold via Steam. I'd assume that this is especially true for independent game makers. So the keys/games are not free review copies. My understanding is that Steam is basically saying "buy through us directly or tough".

[tgb]

TheMix does raise a good point in that the new policy invalidates reviews from people who purchased the game, but did it through GMG or Bundlestars or wherever.

I really don't know how you avoid that.

[Smoove_B]

My understanding is that Steam is basically saying "buy through us directly or tough".

It's more likely that they're saying they know when and how much some one paid for a game because we collected that data. They don't want the review system loaded with comments from free or bundle-cheap copies, so we must filter out non-STEAM purchase reviews. They're doing something a bit similar in the STEAM discussion forums where you can see (easily) when a person is commenting about a game because they put a little icon in the post header that lets you know. In other words, are you communicating with someone that actually owns a copy or a troll account looking to cause problems?

[Rumpy]

They're doing something a bit similar in the STEAM discussion forums where you can see (easily) when a person is commenting about a game because they put a little icon in the post header that lets you know.

So, why not make it more visible, and visible via the reviews? That'd go a long way to improving things, I think.

[Smoove_B]

Maybe that's the ultimate plan. First, purge the non-STEAM purchases. Then add in a sorting system that allows for off-site data collection. I mean, isn't that what this is:

[Carpet_pissr]

I think doing it like Amazon and putting a "Steam Verified Purchase" badge next to a review might be an attractive solution, without having to single out or separate any reviews.

I think even that system is being thwarted now, though, unfortunately. You will see that badge on someone's review and then somewhere at the bottom, it will often show "I received this product in exchange for an unbiased and honest review".

If we could filter those out, it would be a huge help to me personally. Amazon reviews are probably the best source of consumer reviews on the planet right now (at least in the West), and they are in danger of losing that massive driver of sales if they don't get more proactive in fighting the abuse.

(Ironically I am an Amazon Vine member, but I can only speak to my reviews, which are generally exhaustive and well thought out, with no concern about giving a bad review when warranted. I don't see much Vine abuse, but I am sure it exists - I still feel those "I received this product for no cost..." are much higher on the potential scale for abuse though)

[Lorini]

Lorini, you seem to be assuming that the keys were provided for free. My understanding (and I could be completely wrong) is that in some/many cases the keys are not sold via Steam. I'd assume that this is especially true for independent game makers. So the keys/games are not free review copies. My understanding is that Steam is basically saying "buy through us directly or tough".

They no longer allow this. If a dev doesn't post his/her game on Steam they no longer get 'free' keys to give out when it is posted on Steam.

[Lorini]

They're doing something a bit similar in the STEAM discussion forums where you can see (easily) when a person is commenting about a game because they put a little icon in the post header that lets you know.

So, why not make it more visible, and visible via the reviews? That'd go a long way to improving things, I think.

The problem is that people look at the aggregate opinions and don't necessarily bother with the reviews.

[Rumpy]

You will see that badge on someone's review and then somewhere at the bottom, it will often show "I received this product in exchange for an unbiased and honest review".

Well, at least they're honest about it!

But yeah, in general, I think Amazon has cultured a system of trustworthy reviews, and that's powerful. I go looking at reviews there even if I don't end up buying from them.

[Rumpy]

They're doing something a bit similar in the STEAM discussion forums where you can see (easily) when a person is commenting about a game because they put a little icon in the post header that lets you know.

So, why not make it more visible, and visible via the reviews? That'd go a long way to improving things, I think.

The problem is that people look at the aggregate opinions and don't necessarily bother with the reviews.

I think part of the problem on reviews are in the presentation and the number of nonsense that is posted as reviews that quickly overwhelm the legitimate ones. But if something can be done about how they're presented in order to give them more worth in the system itself, I think people would then start to look at reviews.

[Reemul]

You will see that badge on someone's review and then somewhere at the bottom, it will often show "I received this product in exchange for an unbiased and honest review".

Well, at least they're honest about it!

But yeah, in general, I think Amazon has cultured a system of trustworthy reviews, and that's powerful. I go looking at reviews there even if I don't end up buying from them.

Yep Amazon is normally good on checking reviews even if you are getting it elsewhere. But it doesn't take much to start losing in reviews if there is too much abuse.

[TheMix]

Lorini, you seem to be assuming that the keys were provided for free. My understanding (and I could be completely wrong) is that in some/many cases the keys are not sold via Steam. I'd assume that this is especially true for independent game makers. So the keys/games are not free review copies. My understanding is that Steam is basically saying "buy through us directly or tough".

They no longer allow this. If a dev doesn't post his/her game on Steam they no longer get 'free' keys to give out when it is posted on Steam.

I didn't follow this comment at all. If the game isn't on Steam, then the issue is moot. No? And I wasn't talking about free keys. I assumed that developers that sell their games via their own websites as well as Steam are first buying a number of Steam keys - if they are giving out Steam keys. But can't you also add a game to your Steam list without a Steam key? Or play a game outside of Steam that exists on Steam? I understood the primary concern to be that users that are not playing their games via Steam will no longer be able to leave reviews on the Steam page for the game.

[Lorini]

Two recent very popular games, Factorio and Rimworld, both started on the dev's' websites. People who bought those games from the websites were given 'free' keys when the games moved to Steam. That can no longer happen.

Certainly you can buy keys elsewhere or simply attach a game. But if you do a review, your review will not count for that game by default.