mlsanityData to 7 Oct 2026

Trucks Don't Mean Trump: Diagnosing Human Error in Image Analysis

arXiv 2205.07333 ↗First posted 15 May 2022cs.HC cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Algorithms provide powerful tools for detecting and dissecting human bias and error. Here, we develop machine learning methods to to analyze how humans err in a particular high-stakes task: image interpretation. We leverage a unique dataset of 16,135,392 human predictions of whether a neighborhood voted for Donald Trump or Joe Biden in the 2020 US election, based on a Google Street View image. We show that by training a machine learning estimator of the Bayes optimal decision for each image, we can provide an actionable decomposition of human error into bias, variance, and noise terms, and further identify specific features (like pickup trucks) which lead humans astray. Our methods can be applied to ensure that human-in-the-loop decision-making is accurate and fair and are also applicable to black-box algorithmic systems.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 70 references that are in the map. First posted on arXiv 15 May 2022.

Builds on

27 of 70 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  3. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  4. 2018Sanity Checks for Saliency Mapscited by 1,149
  5. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  6. 2017The (Un)reliability of saliency methodscited by 331
  7. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  8. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  9. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  10. 2018Visual Interpretability for Deep Learning: a Surveycited by 155
  11. 2020Learning to Complement Humanscited by 123
  12. 2020Ethical Machine Learning in Health Carecited by 112
15 more
  1. 2013Discrimination in Online Ad Deliverycited by 93
  2. 2014Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discriminationcited by 83
  3. 2020Aligning Superhuman AI with Human Behavior: Chess as a Model Systemcited by 78
  4. 2020A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scorescited by 74
  5. 2019Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootingscited by 67
  6. 2018Direct Uncertainty Prediction for Medical Second Opinionscited by 61
  7. 2021Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowdscited by 58
  8. 2019Discrimination in the Age of Algorithmscited by 51
  9. 2019Roles for Computing in Social Changecited by 49
  10. 2020Interpretable Survival Prediction for Colorectal Cancer using Deep Learningcited by 30
  11. 2021Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systemscited by 30
  12. 2019Learning Representations by Humans, for Humanscited by 18
  13. 2019Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomescited by 15
  14. 2017Fast Threshold Tests for Detecting Discriminationcited by 9
  15. 2018Using Search Queries to Understand Health Information Needs in Africacited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 3.

4 papers.

  1. 2023Domain constraints improve risk prediction when outcome data is missingcited by 8
  2. 2025Tasks and Roles in Legal AI: Data Curation, Annotation, and Verificationcited by 3
  3. 2025Privacy of Groups in Dense Street Imagerycited by 1
  4. 2025Urban Incident Prediction with Graph Neural Networks: Integrating Government Ratings and Crowdsourced Reportscited by 1