, ,

igoranceI ‘ve been writing about the validity of various sorts of wine and food criticism all week, so I might as well continue, since I thought of one more regrettable consequence of crowdsourced reviews.

By “crowsourced reviews” I mean reviews that are the result of the contributions of many people where each contributor adds a small portion to the final outcome. Yelp reviews are an example. It is characteristic of such reviews that no proof of competence or credentials are required to contribute; the wisdom is in the crowd, not in the individuals that make it up. The collective judgment is supposed to be wiser than that the judgment of the individuals.

Although the proponents of crowdsourcing claim that it eliminates the subjectivity involved in more traditional single-person reviews, crowdsourcing has its own biases to contend with. Some biases are well-known and have been widely discussed. There must be some skill or knowledge in the crowd, otherwise the crowd’s judgment can be wildly off-base.  The crowd must be diverse as well. If too many of the same category of person are in the crowd, the results will be skewed toward whatever peculiar characteristic that group possesses. And each participant in the crowd must make an independent judgment. When individual participants base their opinions on the opinions of others in the crowd rather than their own judgment, a bandwagon effect is created, a tendency toward conformity which prevents an accurate account of what the crowd really thinks.

Yelp reviews are subject to all these biases.

But it seems to me there is another bias that is less widely discussed. In the literature on cognitive biases it is called the Dunning-Kruger effect: unskilled individuals typically suffer from delusions about their own superiority. They rate their own ability more highly than is accurate. Or to put the point differently, the unskilled don’t have the tools to recognize their own incompetence. By contrast, skilled people tend to underestimate their ability even when their understanding is good.

How does this influence the wisdom of crowds? I suspect that people who lack knowledge and understanding are more likely to have the confidence to offer their opinion while people with some knowledge may withhold judgment and not participate because they underestimate their ability to contribute. Thus levels of participation are skewed toward those with less skill and understanding.

One factor in favor of individual reviewers is that the individual must take responsibility for what he or she writes. One’s judgment is exposed for all the world to see and that scrutiny makes one a bit more self-critical.  I know that when writing my wine reviews I’m always questioning whether I’m being fair and have considered all the relevant factors, and I have an abiding fear of missing some detail that is distorting my judgment.

Members of crowds bear no such scrutiny. Their judgment is buried in the aggregate results and there are no consequences for being wrong. That is likely to encourage the false confidence uncovered by Dunning and Kruger.

One more reason to lament the decline of individually-sourced reviews.