Wine statistics guru David Morrison has done great work evaluating the degree to which wine scores are consistent between critics and repeatable by the same critic when re-tasting. His data shows a lack of consistency and repeatability.
The main issue, as I see it, is the lack of repeatability of the ratings between tasters. I have previously noted (The poor mathematics of wine-quality scores): Most wine commentators’ wine-quality scores are personal to themselves. That is, the best we can expect from each commentator is that their wine scores can be compared among themselves so that we can work out which wines they liked and which ones they didn’t.
In this recent post he shows that when wines are re-tasted by the same critic there is nevertheless substantial variation in wine scores:
So, about half of the wines were better and half were the same or worse when re-tasted 2 years later, which is what might be expected from random chance. While bottle variation may be a factor here, it is unlikely to change the results (although it might determine which wines did better or worse).
His conclusion, which is almost certainly true, is that wine scores lack the mathematical precision suggested by the use of a quantitative score.
It’s good to have this statistical analysis of wine scores. But we really shouldn’t be surprised by this lack of consistency. It conforms to everything we know and generally acknowledge about wine. Wine grapes change from year to year because of weather variations. Fermentations, barrel ageing and bottle ageing involve constant, unpredictable variation. Even once in the glass, wine continues to change and so do we as we taste it. Our impression of a wine can fluctuate depending on what else we’ve been tasting, and our preferences change over time as well. Wine is inherently a vague object, some of its features difficult to detect even with training. Unlike the clarity of objects directly in our visual field, wine gives us only hints of flavors, scents, and textures all of which can be influenced by environmental factors.
Wine will not sit still for our analysis. Understanding a wine is like tracking a ghost through fog. It’s the variation that makes wine interesting.
Anyone who has spent much time around wine knows all of this already. So why would anyone expect wine critics to be consistent? Why would we want them to be consistent? The demand for consistency and precise judgment violates everything we know about wine. Yet that demand never goes away.
In this wine, is not alone. Our civilization is based on the idea that behind the churn and flux of life there must be something stable and unchanging. Despite evidence to the contrary we still can’t get free of that assumption.