• About
  • American Foodie: Taste, Art and the Cultural Revolution
  • Mindful Eating 2
  • Our Wine Review Philosophy

Edible Arts

~ Exploring the Aesthetics and Philosophy of Food and Wine

Edible Arts

Tag Archives: replication crisis

Why There is a Crisis in Science

12 Monday Aug 2019

Posted by Dwight Furrow in Philosophy

≈ Leave a comment

Tags

replication crisis, science, scientific method

scientist in a labThis is a bit of a detour from the wine and food topics on which I usually write. But it is an important issue that concerns the credibility of science. The credibility of science and expertise in general are now coming under constant attack. Yet no advanced technological society can survive for long without a science that is both accurate and is believed to be accurate by the public that depends on it. Attacks on science threaten the very foundation of modern societies. Yet the skepticism about science is not coming only from opportunistic politicians or whack-job nut cases. There is good reason to be skeptical of our scientific institutions and how they function because of the replication crisis that is beginning to effect all scientific disciplines.

The “replication crisis” refers to the fact that an increasingly large number of studies is the sciences and the social sciences can’t be replicated. When independent researchers try to repeat a study they come up with vastly different results that contradict the results of the original study. As Aubrey Clayton notes in a very useful article in Nautilus:

An analysis of preclinical cancer studies found that only 11 percent of results replicated; of 21 experiments in social science published in the journals Science and Nature, only 13 (62 percent) survived replication; in economics, a study of 18 frequently cited results found 11 (61 percent) that replicated; and an estimate for preclinical pharmacology trials is that only 50 percent of the positive results are reproducible, a situation that, given the immense size of the pharma industry, has been estimated to cost labs something like $28 billion per year in the U.S. alone.

This is not good. Clayton’s article is important because he explains why there is a replication crisis—it has to do with a long simmering debate in statistical analysis. I’m no expert in statistics but I will try to give a clear summary of what this debate is about—the fate of civilization may depend on it. Clayton uses several examples; I will focus on one of them.

Suppose an otherwise healthy woman in her forties notices a suspicious lump in her breast and goes in for a mammogram. The report comes back that the lump is malignant. She wants to know the chance of the diagnosis being wrong. Her doctor answers that, as diagnostic tools go, these scans are very accurate. Such a scan would find nearly 100 percent of true cancers and would only misidentify a benign lump as cancer about 5 percent of the time. Therefore, the probability of this being a false positive is very low, about 1 in 20.

This is an approach to statistics that uses significance testing to determine the validity of a study.

Suppose we scan 1 million similar women, and we tell everyone who tests positive that they have cancer. Then, among those who actually have cancer, we will be correct every single time. And among those who don’t have it, we will be only be incorrect 5 percent of the time. So, overall our procedure will be incorrect less than 5 percent of the time.

The claim about validity is based on how often a patient would test positive if the condition were absent. In other words the likelihood that there is no correlation between the lump and malignancy is less than 5%, meaning the result is statistically significant.

But as Clayton notes there is a problem here. The study doesn’t take into account the background rate of cancer among women with a suspicious lump.

For the breast cancer example, the doctor would need to consider the overall incidence rate of cancer among similar women with similar symptoms, not including the result of the mammogram. Maybe a physician would say from experience that about 99 percent of the time a similar patient finds a lump it turns out to be benign. So the low prior chance of a malignant tumor would balance the low chance of getting a false positive scan result. Here we would weigh the numbers:

(0.05) * (0.99)              vs. (1) * (0.01)

We’d find there was about an 83 percent chance the patient doesn’t have cancer.

According to this analysis the woman in the example was likely misdiagnosed. The replication crisis is happening because many studies are finding a relationship between phenomena that does not exist. Thus, researches fail to duplicate it.

The problem according Clayton is that we should be including in our calculations, the prior probability of a theory before making an observation to test the theory. This is called Bayesian probability theory—it has been around for decades but has not been widely accepted.

Why hasn’t it been widely accepted?

The main reason scientists have historically been resistant to using Bayesian inference instead is that they are afraid of being accused of subjectivity. The prior probabilities required for Bayes’ rule feel like an unseemly breach of scientific ethics. Where do these priors come from? How can we allow personal judgment to pollute our scientific inferences, instead of letting the data speak for itself?

For Baysians, we have to assign a probability before making the observations and then we allow the observations to influence the initial probability estimate. But that initial assignment is only going to be an educated guess—like the physician’s judgment above based on her experience that most lumps are benign. Might such an all encompassing fear of subjectivity be irrational?

This may have a happy ending. As Clayton notes:

Medical students are now routinely taught the diagnostic importance of base incidence rates. Bayes’ theorem helps them properly contextualize test results and avoid unnecessarily alarming patients who test positive for something rare.

Thankfully, science when done honestly is a self-correcting practice, and now that the replication crisis is well known scientists are responding by taking a harder look at statistical relationships that seem implausible. But an awful lot of bad science has already entered the mainstream giving aide and comfort to charlatans, quacks, and opportunistic politicians who would love to ignore inconvenient facts. I hope the self-correction is not too late.

Search

Join the Food Revolution

American Foodie: Taste, Art, and Cultural Revolution

Get Our Guide to Wine Tasting in San Diego

Wine Guide to San Diego at Amazon

Categories

  • 3 Quarks Daily Column
  • Amuse Bouche
  • Aphorisms
  • Art and Food
  • Art and Wine
  • Contemporary Food Culture
  • Daily Blog Summary
  • Edible Art
  • Food History
  • Food Science
  • General Aesthetics
  • Monthly Newsletters
  • Philosophy
  • Philosophy of Food and Wine
  • Recipes
    • Appetizer
    • Bread
    • Dessert
    • Lunch
    • Main Dish
    • Salad
    • Side Dish
    • Soup
  • Restaurant Reviews
  • Roving Decanter Posts
  • San Diego Wine Events
  • San Diego Wines
  • The Art of Wine
  • The Creative Kitchen
  • The Ethics of Eating
  • Travel
  • Uncategorized
  • Wine Culture
  • Wine Reviews
    • Ageing Report
    • Budget Wines
  • Winemaker Interviews
  • RSS - Posts
  • RSS - Comments

Visit Us On Facebook

Edible Arts

Promote Your Page Too

Like Us On Facebook

Like Us On Facebook

Enter your email address to follow this blog and receive notifications of new posts by email.

Blogroll

  • Contemporary Aesthetics
  • egullet
  • Fermentation
  • Ideas in Food
  • Jamie Goode
  • Leite's Culinaria
  • Mindful Eating 2
  • Orangette
  • San Diego Wine and Food
  • Serious Eats
  • Smitten Kitchen
  • The Curious Cook
  • The Philosophy of Food Project
  • The Wine Elite
  • Vinography
  • Zester Daily

aesthetics aesthetics of food aesthetics of food and wine aesthetics of wine Aphorisms Argentinian wine art and food Art and wine Australian Wines blogging Bordeaux Cabernet Franc Cabernet Sauvignon California wine California Wines Chardonnay Chianti Chilean Wine Columbia Valley creativity food aesthetics food and emotions food and identity food and wine aesthetics food business French wine Italian Wine Italian Wines locavorism Malbec Matt Kramer merlot Monterey County natural wine objectivity objectivity and taste oregon wines Paso Robles Petite Sirah Philosophy of food Philosophy of food and wine philosophy of wine pinot noir Red blend Red blends restaurants Rhone Blends Riesling San Diego Wines Sangiovese Santa Barbara Wines Sauvignon Blanc Sonoma Wines Spanish Wine Spanish Wines Sparkling wine Steve Heimoff Syrah Tempranillo terroir Washington State Wines Willamette Valley wine aesthetics wine and art wine and emotions wine blogs wine business wine criticism wine education wine evaluation wine news wine science wine tasting Wine Writing zinfandel

Networked Blogs

NetworkedBlogs
Blog:
Edible Arts
Topics:
Food, Aesthetics, Wine
 
Follow my blog

Create a free website or blog at WordPress.com.

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy