The Problem With a Five Star Review

[Josh Burch]

When we review products reviewers seem to give out five star reviews with reckless abandon. I've been trying to wrap my head around their reasoning but I still can't figure out why it is so popular and acceptable to give the highest review possible for ever single product on the market.

-When a restaurant has 5 Michelin stars it is considered to be one of the best restaurants in the world and is distinguished as being the most elite of elite.
-Only the most luxurious hotels can get 5 star ratings.
-Critics rarely give 5 star ratings to movies so that you can clearly tell what is the best.

Why is it that when we look on almost any review site that they are almost overflowing with five star reviews? When is three stars going to be the average, four stars for great products and five stars only for those products that no magician can live without?

 

[Tower of Lunatic Meat]

As far as this site goes, I think there's a couple problems with the star rating system:

1) There needs to be at least a 50 word minimum for giving any sort of review whatsoever. I think this would cut down on a lot of the negative/positive 'fluff' ratings that are passed out so freely.

2) For written reviews, I think a lot of people (myself included) feel that only reviews that promote a product are only posted and given points for. Which is why I think a lot of people just give up and go right for the star ratings. Why write something if there's a chance it's going to get rejected? I wrote a good 4 star review for a product--3 times. It finally got accepted when I wrote the same thing again, but gave it 5 stars. I felt that was a bit off. I understand the need for quality control. But there needs to be consistency as there are reviews that are 7 word sentences. They got Elite Points for that.

 

[CaseyRudd]

As far as this site goes, I think there's a couple problems with the star rating system:

1) There needs to be at least a 50 word minimum for giving any sort of review whatsoever. I think this would cut down on a lot of the negative/positive 'fluff' ratings that are passed out so freely.

2) For written reviews, I think a lot of people (myself included) feel that only reviews that promote a product are only posted and given points for. Which is why I think a lot of people just give up and go right for the star ratings. Why write something if there's a chance it's going to get rejected? I wrote a good 4 star review for a product--3 times. It finally got accepted when I wrote the same thing again, but gave it 5 stars. I felt that was a bit off.

Since users can receive points for submitting reviews, we often get a flood of reviews from the same user looking to receive free points for products they likely do not own. When we see 3+ reviews in a row from the same user we assume they are looking for free points and not trying to give an honest review. Obviously this is not the case in some instances; there are some users who do own all of the products they are reviewing at one time. It just seems suspicious from our end seeing a ton of reviews from the same user on a variety of products. Getting a review published has nothing to do with the positive or negative rating; as long as the review is constructive and is written in a professional manner it should get published, regardless of rating. You can give a product a 2 star review and explain why you gave it that review without using the words "the design sucks so I'm giving it a 2 star rating". If you submit reviews separate from each other, like 1 every day or every other day (only if you own said product) then your review should get posted if it meets the guidelines.

Our guidelines for getting a review posted:

1. The review is professionally written with proper spelling, grammar, and punctuation. Things like "i ---LUV--- magic!<,1" will be declined.

2. Whether positive or negative, the review is respectful to the artist or creator. If you like it or hate it, we would love to hear about it. Just be respectful to the people who created the trick.

3. The review provides meaningful insight into the product in reference and is factually accurate and not misleading or confusing. If your review is simply "I like this", it brings no meaningful insight, so it will be declined. Likewise, if you attempt to claim (falsely) that one of our tricks has the same method as a different trick, it will also be declined.

 

[Steven Lafitte]

When we review products reviewers seem to give out five star reviews with reckless abandon. I've been trying to wrap my head around their reasoning but I still can't figure out why it is so popular and acceptable to give the highest review possible for ever single product on the market.

-When a restaurant has 5 Michelin stars it is considered to be one of the best restaurants in the world and is distinguished as being the most elite of elite.
-Only the most luxurious hotels can get 5 star ratings.
-Critics rarely give 5 star ratings to movies so that you can clearly tell what is the best.

Why is it that when we look on almost any review site that they are almost overflowing with five star reviews? When is three stars going to be the average, four stars for great products and five stars only for those products that no magician can live without?

Part of the issue with that is all comparisons between Hotel & Restaurant reviews and Magic reviews stop after the basic assumption that the higher the ranking obviously means (or is suppose to mean) the better the service/quality/effect.

There are actual star rating systems in place for Hotels & Restaurants such as national consumer travel organizations that assign star ratings to hotels. Michelin relies on a staff of anonymous restaurant reviewers and then all of the reviewers come together to discuss and decide on which restaurants will be awarded the stars.

Magic on the other hand has nothing like this whatsoever, because there is no official set of criteria to go by unlike those of Hotels & Restaurants. One persons 2 might be another persons 5 depending on what other products they've tried before and what their expectations are, therefore, subjectivity is going to play a big role and that can't be helped.

|| Steven

 

[Josh Burch]

I guess I see the same pattern with Amazon Reviews as well. It seems like in order to tell everyone that the product is good they automatically default to a five star review as to avoid confusion. Unfortunately this skews the review system to be overwhelmingly positive.

 

[Bryant_Tsu]

I think the problem is that there are many different types of consumers out there. For cards I think the rating system is fine, it all comes down to personal opinion on design (most t11 decks handle similarly anyway). When you get to the tricks section you start to see different types of consumers. You may have the children ages 8-12 who look for simple tricks and love them, no matter the cons (because that's what their current limit may be) and therefore give a five star review. Amateurs or enthusiasts may see the same trick and give it a lower rating or find a pretty good trick and give that a five star rating because that's as far as they are willing to progress into the realm of magic and cardistry. As for professionals their 4-5 star choices may be very few (like what you describe). These people do magic for a job and therefore must be very discerning of what tricks they should perform. Overall, the professionals may feel like there are too many 5 star ratings (which may be half true). However at the end of the day one must realize that theory11's consumer base is comprised mostly of enthusiasts like myself, and therefore the ratings work more towards that group to help find whats good or pretty good.

Perhaps a rule of thumb for the professionals out there, lower the rating a star or two and base it off of that.

-Bryant

 

[Tower of Lunatic Meat]

3. The review provides meaningful insight into the product in reference and is factually accurate and not misleading or confusing. If your review is simply "I like this", it brings no meaningful insight, so it will be declined. Likewise, if you attempt to claim (falsely) that one of our tricks has the same method as a different trick, it will also be declined.

On the other side of the coin; while the reviews chosen are respectful to the artist, I don't see how these examples steer potential customers towards a purchase (all of these are on the front page of The Wire):

'Pretty interesting flourish and it's fun to play around with.'

'Excellent trick and really powerful effect.'

'It's a very cool effect with an excellent price. Good job.'

'Great effect; not very easy but was great to learn. Got to use this at Starbucks one day!'

 

[Justin.Morris]

Interesting thought Josh, I wonder if there is a better rating system that would better reflect attitudes. Like a rating out of 3 (like, neutral, dislike) and a reason why. That way it's not about sort of good vs really good, but more about the reasons why people like it.

 

[Tower of Lunatic Meat]

Interesting thought Josh, I wonder if there is a better rating system that would better reflect attitudes. Like a rating out of 3 (like, neutral, dislike) and a reason why. That way it's not about sort of good vs really good, but more about the reasons why people like it.

Just remembered this. The Air Force rating system was (not sure if the new system before I got out fixed it entirely) VERY similar to the 5 star rating system. Seriously. Except in that case, getting a '4' could result in not making higher ranks way down the road (if you had a '4' within 10 years of testing for E8. Guess who's not making E8 that year).

In the case of products, all I can think of is people being afraid to give a bad rating in case it really works for someone else. Or they are afraid of making a bad judgment call and being lambasted for it.

I also think part of the problem is that a lot of people aren't willing to write a justification for WHY they feel that way. Because that would be work. Either out of fear or laziness; both are natural enemies of producing better products and upgrades.

 

[ChrisWiens]

Interesting thought Josh, I wonder if there is a better rating system that would better reflect attitudes. Like a rating out of 3 (like, neutral, dislike) and a reason why. That way it's not about sort of good vs really good, but more about the reasons why people like it.

Intriguingly we have a Games Magacine here in Europe (Eurogamers.net) which recently changed its rating system from a ten point system to a simple four point system, which is a good choice in my opinion.
IMO the explaination WHY I liked or didn't like a product should be more important. It forces you to read the review instead of just looking for ratings. It offers a better context to weigh up the arguments for yourself without the influence of a rating system.

 

[Colin]

I think you will see this pattern any where that allows reviews to be written by the general public as opposed to places that offer more professional critique. I have written book reviews of newspapers for years now and can count on one hand the number of '5 star reviews' I have published and it is not for lack of a lot of good reads. From a professional stand point to give something 5 stars or a 10/10 it has to be flawless and very few things are. There are a lot of very good things out there but perfect is rare.

Posting on websites though doesn't require any kind of background knowledge or justification for the rating any many of them will simply be based on immediate emotional reaction to the product. With magic you have likely just learned the secret to the new coolest thing you've ever seen so of course it deserves a quick 5 star rating or it turns out it takes actual practice and involves some setup so Booo 1 star when if you were to sit down with the right experience and knowledge and took into account production value, teaching method, proper crediting and all the things that go into teaching a magic effect the answer is probably somewhere in the middle.

For me the best way to use reviews that are written by the public is to read a lot of them so that you become familiar enough with the community so that you know when they start to rave about something if you are likely to want to check it out further.