I certainly share Anne Burchett’s complaint about the constant requests for reviews we’re asked to submit for everything from books to Instacart deliveries:
For wine as for everything else, peer recommendations are where it’s at now: no more arrogant sommeliers dictating your choices, no more stuffy experts. Some clever clogs cottoned on to the fact humans are more likely to listen to the advice of an approachable friend or a fellow random consumer than to that of an intimidating stranger however more qualified they may be, and the Internet did the rest. But how well does it work for wine?
Not well as Burchett argues. Most customer reviews on Vivino and similar rating sites are about as informative as “superb” or “tastes great.” And even when you stumble upon a more informative review, you know nothing about that person’s skills, credentials, track record, or approach to their rating system.
She points out the irony in all of this:
There is an implied level of trust in the review system that sits at odds with society’s current cynicism. You will buy the shoes I recommend but you won’t listen to a scientist telling you to get a vaccine?
Burchett focuses on how little we can know from individual reviews. But the whole phenomenon of crowdsourced reviews, which the rating system employs, is even worse when it comes to credibility.
Although the proponents of crowdsourcing claim that it eliminates the subjectivity involved in more traditional expert reviews, crowdsourcing has its own biases. There must be some skill or knowledge in the crowd, otherwise the crowd’s judgment can be wildly off-base. The crowd must be diverse as well. If too many of the same category of person are in the crowd, the results will be skewed toward whatever peculiar characteristic that group possesses. And each participant in the crowd must make an independent judgment. When individual participants base their opinions on the opinions of others in the crowd rather than their own judgment, a bandwagon effect is created, a tendency toward conformity which prevents an accurate account of what the crowd really thinks.
Customer review sites such as Vivino are subject to all these biases.
But it seems to me there is another bias that is less widely discussed. In the literature on cognitive biases, it is called the Dunning-Kruger effect: unskilled individuals typically suffer from delusions about their own superiority. They rate their own ability more highly than is accurate. Or to put the point differently, the unskilled don’t have the tools to recognize their own incompetence. By contrast, skilled people tend to underestimate their ability even when their understanding is good.
How does this influence the wisdom of crowds? I suspect that people who lack knowledge and understanding are more likely to have the confidence to offer their opinion while people with some knowledge may withhold judgment and not participate because they underestimate their ability to contribute. Thus, levels of participation are skewed toward those with less skill and understanding.
One factor in favor of individual reviewers is that the individual must take responsibility for what he or she writes. One’s judgment is exposed for all the world to see and that scrutiny makes one a bit more self-critical. I know that when writing my wine reviews I’m always questioning whether I’m being fair and have considered all the relevant factors, and I have an abiding fear of missing some detail that is distorting my judgment.
Members of crowds bear no such scrutiny. Their judgment is buried in the aggregate results and there are no consequences for being wrong. That is likely to encourage the false confidence uncovered by Dunning and Kruger.
Yet, as Burchett points out, customer reviews are not going away. Amazon likes them and what Amazon likes we must live with.