The Nonsense of Customer Reviews

5 star reviewI certainly share Anne Burchett’s complaint about the constant requests for reviews we’re asked to submit for everything from books to Instacart deliveries:

For wine as for everything else, peer recommendations are where it’s at now: no more arrogant sommeliers dictating your choices, no more stuffy experts. Some clever clogs cottoned on to the fact humans are more likely to listen to the advice of an approachable friend or a fellow random consumer than to that of an intimidating stranger however more qualified they may be, and the Internet did the rest. But how well does it work for wine?

Not well as Burchett argues. Most customer reviews on Vivino and similar rating sites are about as informative as “superb” or “tastes great.” And even when you stumble upon a more informative review, you know nothing about that person’s skills, credentials, track record, or approach to their rating system.

She points out the irony in all of this:

There is an implied level of trust in the review system that sits at odds with society’s current cynicism. You will buy the shoes I recommend but you won’t listen to a scientist telling you to get a vaccine?

Burchett focuses on how little we can know from individual reviews. But the whole phenomenon of crowdsourced reviews, which the rating system employs, is even worse when it comes to credibility.

Although the proponents of crowdsourcing claim that it eliminates the subjectivity involved in more traditional expert reviews, crowdsourcing has its own biases. There must be some skill or knowledge in the crowd, otherwise the crowd’s judgment can be wildly off-base.  The crowd must be diverse as well. If too many of the same category of person are in the crowd, the results will be skewed toward whatever peculiar characteristic that group possesses. And each participant in the crowd must make an independent judgment. When individual participants base their opinions on the opinions of others in the crowd rather than their own judgment, a bandwagon effect is created, a tendency toward conformity which prevents an accurate account of what the crowd really thinks.

Customer review sites such as Vivino are subject to all these biases.

But it seems to me there is another bias that is less widely discussed. In the literature on cognitive biases, it is called the Dunning-Kruger effect: unskilled individuals typically suffer from delusions about their own superiority. They rate their own ability more highly than is accurate. Or to put the point differently, the unskilled don’t have the tools to recognize their own incompetence. By contrast, skilled people tend to underestimate their ability even when their understanding is good.

How does this influence the wisdom of crowds? I suspect that people who lack knowledge and understanding are more likely to have the confidence to offer their opinion while people with some knowledge may withhold judgment and not participate because they underestimate their ability to contribute. Thus, levels of participation are skewed toward those with less skill and understanding.

One factor in favor of individual reviewers is that the individual must take responsibility for what he or she writes. One’s judgment is exposed for all the world to see and that scrutiny makes one a bit more self-critical.  I know that when writing my wine reviews I’m always questioning whether I’m being fair and have considered all the relevant factors, and I have an abiding fear of missing some detail that is distorting my judgment.

Members of crowds bear no such scrutiny. Their judgment is buried in the aggregate results and there are no consequences for being wrong. That is likely to encourage the false confidence uncovered by Dunning and Kruger.

Yet, as Burchett points out, customer reviews are not going away. Amazon likes them and what Amazon likes we must live with.


  1. An extremely complex subject, Dwight. On the one hand, we have Emerson suggesting that “To believe your own thought, to believe that what is true for you in your private heart is true for all men, — that is genius.” And who can argue with the leader of transcendentalism?

    Well, lots of people. Dr. Fauci, to use a recent and somewhat politically charged example.

    But the appeal of these crowd sourced reviews is simple: they use language and cultural references that clearly communicate something to their social cohort. “This wine is TOTAL LADY GAGA” may not communicate much to me, but it certainly might to a fan of that singer. The same may not be true of the vaunted expert, whose use of language and references may highlight or even be chosen to stress how far removed that expert is from their social cohort. We’ve all seen those reviews…

    I find social media reviews helpful only if taken larger doses. One review on social media is, as you suggest, often not helpful. But if the hotel I am considering is rated five stars because it is a party all night long, I know that the hotel is the last place I want to stay. And that is very helpful.

    Experts who review wine have their own biases, of course. A single review from an expert rarely convinces me of anything–only when I have read a number of reviews and fully understand the opinions and biases of the reviewer will I use those reviews to influence a purchase. And as often as not, the influence will negative, as I have discovered by reading the reviews that the expert prefers wines that are riper and more powerful than I like to drink.

    And of course, the premise of all this is that consumers buy a wine because of the flavor. That in itself is inaccurate. Consumers often buy a wine to express their personal style and taste, and to belong to a certain social cohort. And in that case, those social media reviews are pure gold.

  2. Hi Paul,
    Actually, “this wine is totally Lady Gaga” would give me a lot of information if I were confident the person making the claim was adept at making pop cultural references. That’s the problem with reviews from the public–you don’t know. At least with an expert, you can look at a track record. I agree about hotels. I find crowdsourced reviews helpful for some commodities because there is lots of agreement about what makes a good basic hotel room. Quiet, no bed bugs, etc.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.