All of my product, movie, concert, game and music reviews are written long-form, and never have any numerical rating associated with them.
They almost did. I had this clever idea to have a 5 star scale and have the rating system be glasses. Each pair of glasses would equal a star, and a monocle would equal half a star. However I sacked the idea.
Why did I change my mind?
Joe was one of my undergraduate professors, and learning from him was one of those rare life-changing teacher experiences. But I digress.
Why 1 – 5 Scales Suck
His classes were unconventional in a lot of ways, but the most significant example was his strong opposition to generic scale rating systems. This didn’t make him popular in the social sciences because 1 – 5 rating systems are pretty much the standard means for measuring any human experience in psychology, sociology, political science, and communication.
He illustrated this very simply in what turned out to be the most memorable five minutes I had in two decades of school.
Joe began a class by instructing us to take out a piece of paper, and create four 1 – 5 scales on it. Then he asked the following four questions and had us answer on those scales:
- How big is the Moon?
- How big is the Sun?
- How big is a penny?
- How big is a dime?
The answers were pretty much along these lines:
- How big is the Moon? – 5
- How big is the Sun? – 5
- How big is a penny? – 1
- How big is a dime? – 1
- How big is the Moon? – 4
- How big is the Sun? – 5
- How big is a penny? – 2
- How big is a dime? – 1
The problem was immediately apparent to everyone with a pulse. According to this research, students perceptions are that the Moon and Sun are the same size, while pennies and dimes are the same size. The kicker being that the Moon and the Sun are five times larger than pennies and dimes.
His point was that these methods of measurement are meaningless because the ratings have no context, and there’s no opportunity for logical mathematical comparison. So he (and a few others) created a system of measure that was based on comparison. It’s complex, and I’m not going to get into it here.
Why 1 – 100 Scales Suck
So you might be asking yourself, why not use a 1 – 100 scale?
It still has no comparison value, but mostly I don’t know the difference between an 83 and an 84 in terms of quality. Whenever I see ratings like 93, I find myself completely baffled by how someone came up with that number. Sometimes I reach the bottom of a review and see some strange number and I actually laugh as I imagine my eighth grade algebra teacher yelling, “Show your damn work!”
The bottom-line
I don’t know how to boil my complex thoughts about something into an arbitrary number. Plus, if I write a number it devalues all of the rest of the thought and nuance that went into the review.
Read a review, don’t read a review. Numerical rating systems distort reality. I won’t be using them.
The Geek Whisperer 