Biases in Crowdsourced Reviews

crowdsourcingI’ve found crowdsourced review sites like Yelp to be utterly unreliable for anything but the most basic information about a restaurant. A new study suggests that a number of biases influence people who write such reviews.

The size and racial makeup of a city, the price of a meal and even the weather can skew the quality and quantity of online restaurant reviews, according to the first large-scale academic study to analyze how outside factors affect crowd-sourced review sites.

The study, which will be released Wednesday, used computer models to examine nearly 1.1 million reviews of 840,000 restaurants over nearly a decade.

Among the findings:

— Reviews written in the summer were more likely to be negative

— Reviews written when temperatures were below 40 degrees or above 100 degrees or when snowing or raining  were more likely to be negative.

— Restaurants in the Northeast and on the West Coast were reviewed more than those in the South or the Midwest.

— People living in urban areas were more likely to write reviews

—  Diners in large cities were more forgiving about waiting times that those in smaller communities

— Sushi restaurants were given higher reviews than burger restaurants suggesting ambiance and price influenced the results.

One limitation of the study is that Yelp reviews were not included in the results.

Many of these results suggest that a person’s mood influences their judgment–not a surprising result at all.

But when this evidence is combined with an earlier study that suggests crowdsourced reviewers are influenced by what other reviewers have already written, especially when the reviews are positive, there is reason to doubt the reliability of crowdsourced review sites.

There is still no substitute for a well-trained, experienced professional critic. Yes, they are subject to biases like anyone else, but professional reviewers, whose livelihood depends on their reliability, are more likely to be aware of that and make a conscious effort not to let biases related to mood or peer pressure influence them.

One comment

  1. I completely agree. I have much more confidence in professional reviews. Professionals know how to be objective. NPR had a story several months ago about how some crowdsource sites (Yelp was specifically named) take payments from businesses to sort reviews with the positives at the top.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.