Thread: What would be better than sorting by "Average Star Rating" ?

Hi

My client is looking for a ranking system for his products (which have been reviewed) that is more useful than "Average Star Rating".

As he correctly points out, the problem with "Average Star Rating" is that a very low numbers of ratings it becomes totally unreliable. For example suppose we had 100 reviews averaging 4.8 stars, surely that is FAR better then 1 review of 5 stars, but it's Average star rating of "4.8" is obviously lower than the one review whose average is "5" !

Given that the user does not know nothing about any of the reviewers, and thus can not really trust any of them individually, what is the logical solution of number of reviews vs Star Ratings?

i.e. What is the best sort order to display reviewed products in ?

Because sorting by "Average Star Rating" clearly doesnt make any sense (at low numbers of reviews).

With thanks


John

P.S. I am not a mathematician (I am a humble web designer), but I am hoping that there is simple solution to this problem.

Originally Posted by Eric69

Hi

My client is looking for a ranking system for his products (which have been reviewed) that is more useful than "Average Star Rating".

As he correctly points out, the problem with "Average Star Rating" is that a very low numbers of ratings it becomes totally unreliable. For example suppose we had 100 reviews averaging 4.8 stars, surely that is FAR better then 1 review of 5 stars, but it's Average star rating of "4.8" is obviously lower than the one review whose average is "5" !

Given that the user does not know nothing about any of the reviewers, and thus can not really trust any of them individually, what is the logical solution of number of reviews vs Star Ratings?

i.e. What is the best sort order to display reviewed products in ?

Because sorting by "Average Star Rating" clearly doesnt make any sense (at low numbers of reviews).

With thanks


John

P.S. I am not a mathematician (I am a humble web designer), but I am hoping that there is simple solution to this problem.

The reviewer who gave the product rating of 1 star is just a valid as the review who gave 5.
If you are going to use "average" then individual ratings should not be considered.
Could you give an example of the acceptable/unacceptable rating or sort you have at the moment?

Because sorting by "Average Star Rating" clearly doesnt make any sense (at low numbers of reviews).

BUT a single individual's rating does?

Originally Posted by aidan

The reviewer who gave the product rating of 1 star is just a valid as the review who gave 5.
If you are going to use "average" then individual ratings should not be considered.
Could you give an example of the acceptable/unacceptable rating or sort you have at the moment? BUT a single individual's rating does?

Sorry but I think you have slightly misunderstood my point.

Try this: Suppose three people give a product a 5 star rating. Their average rating would thus be "5.0 stars". However in the real world, we know that nothing is perfect, (and more to the point not all reviewers are ever likely to agree that something is perfect!) and thus the three reviews all being 5 stars, can be seen as a "statistical blip" (in layman's language).

Now suppose 10,000 people rated another product an average of 4.9 stars.

Well, surely it feels wrong to give this 'accidental' maximum possible score (of 5.0 stars - average) a higher priority than a more informed consensus view formed by the 10,000 users.

It seems to me that although average star raing works well at higher numbers as a useful metric where lots of opinions are being collected, it becomes more and more unreliable... and thus less and less useful a metric the fewer opinions are involved.

I find it hard to know what the ideal system should behave, other than being broadly based on average number of ratings but punishing the lower numbers of ratings.

In fact I bounce it back to the reader of this thread: Which would you prefer to see given higher priority?
a) Three users giving an average of 5.0 star rating, or
b) 10,000 average of 4.9 stars.

It would seem to me that someone somewhere is bound to hate whatever is being rated so it is almost certain that 10,000 people would almost never average 5.0 stars.

And if you choose "b)", then what about 10,000 users averaging 4.8 stars, or 4.7 stars... in fact at what point do the two equate?
i.e. exactly how many average stars by 10,000 people would equate to 3 people averaging 5 stars?

As a non-mathematician myself, my guess is that it would depend partly on how reliable you thought the reviewers are likely to be in this instance... but supposing you simply have no idea whatsoever how reliable the ratings are (e.g. the reviews on Amazon)... is there any general solution to the problem? What does the rational person do?

Anyhow as a total non-mathematician I just wondered if anyone had formally solved this problem!

Cheers


John

I think you have slightly misunderstood my point.

You're wrong. I have completely misunderstood your point.

You are upset because a subset of the 10003 reviewers gave a perfect "5-star" rating to the product.
If you do not believe/accept the results of those who review product,
then you should substitute your own review in place of those first few (3, or 10, or 30, or up to the number you feel is ok, maybe 10004).

I just wondered if anyone had formally solved this problem!

with all due respect, could you formally state the problem?

I agree, Best way around your pbpoblem is not to have any rating appear until x number of ratings have been submitted. Or perhaps show a small barchart with the frequency of each rating point 1-5.

Originally Posted by aidan

You're wrong. I have completely misunderstood your point.

You are upset because a subset of the 10003 reviewers gave a perfect "5-star" rating to the product.
If you do not believe/accept the results of those who review product,
then you should substitute your own review in place of those first few (3, or 10, or 30, or up to the number you feel is ok, maybe 10004).

with all due respect, could you formally state the problem?

@Aidan,
You are coming at this from a direction I that I find strange probably because I do not have and adequate background in mathematics or possibly because I am simply clever enough to understand. (Either that or you are just being weird for reasons of your own - who knows)

No I am not actually 'upset' by anything.
I am simply trying to establish how we should respond if we are being purely rational to data that contains varying numbers of ratings and individual rating values.

Given that we only partially trust each of the reviewers, if we ignore their actual review words and focus purely on the ratings themselves,
I seek an equation that would put such data into a sort order.

Is that formal enough for you?

Originally Posted by Niall101

I agree, Best way around your pbpoblem is not to have any rating appear until x number of ratings have been submitted. Or perhaps show a small barchart with the frequency of each rating point 1-5.

Sorry but, IMHO, not showing any ratings until x number of ratings is a total cop-out (I thought this was a math forum not a let's have rough guess forum! ) Okay, let's assume you are correct. What should that number x be - and why?

Yes, I do agree that a small bar chart is extremely helpful when looking at any individual product (e.g. this is what they do on Amazon).

BUT this is still missing the point. The point is that, given that the user is looking for the best possible product, what is the best way of sorting products, - each of which has its own frequency of ratings bar chart?
(whereby the products that are most likely to be the best products are listed first). And what is the formula for this?

Does that make sense?