# Trucks Don't Mean Trump: Diagnosing Human Error in Image Analysis

arXiv: https://arxiv.org/abs/2205.07333 · HTML: https://mlsanity.com/paper/2205.07333/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 70 references that are in the map.
- First posted on arXiv 15 May 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Visual Interpretability for Deep Learning: a Survey](https://mlsanity.com/paper/1802.00614/) (2018)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [Ethical Machine Learning in Health Care](https://mlsanity.com/paper/2009.10576/) (2020)
- [Discrimination in Online Ad Delivery](https://mlsanity.com/paper/1301.6822/) (2013)
- [Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination](https://mlsanity.com/paper/1408.6491/) (2014)
- [Aligning Superhuman AI with Human Behavior: Chess as a Model System](https://mlsanity.com/paper/2006.01855/) (2020)
- [A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores](https://mlsanity.com/paper/2002.08035/) (2020)
- [Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings](https://mlsanity.com/paper/1904.01596/) (2019)
- [Direct Uncertainty Prediction for Medical Second Opinions](https://mlsanity.com/paper/1807.01771/) (2018)
- [Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowds](https://mlsanity.com/paper/2105.06496/) (2021)
- [Discrimination in the Age of Algorithms](https://mlsanity.com/paper/1902.03731/) (2019)
- [Roles for Computing in Social Change](https://mlsanity.com/paper/1912.04883/) (2019)
- [Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems](https://mlsanity.com/paper/2102.04201/) (2021)
- [Interpretable Survival Prediction for Colorectal Cancer using Deep Learning](https://mlsanity.com/paper/2011.08965/) (2020)
- [Learning Representations by Humans, for Humans](https://mlsanity.com/paper/1905.12686/) (2019)
- [Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomes](https://mlsanity.com/paper/1904.02095/) (2019)
- [Fast Threshold Tests for Detecting Discrimination](https://mlsanity.com/paper/1702.08536/) (2017)
- [Using Search Queries to Understand Health Information Needs in Africa](https://mlsanity.com/paper/1806.05740/) (2018)

## Built on by (most cited first, 4 of 4)

- [Domain constraints improve risk prediction when outcome data is missing](https://mlsanity.com/paper/2312.03878/) (2023)
- [Tasks and Roles in Legal AI: Data Curation, Annotation, and Verification](https://mlsanity.com/paper/2504.01349/) (2025)
- [Privacy of Groups in Dense Street Imagery](https://mlsanity.com/paper/2505.07085/) (2025)
- [Urban Incident Prediction with Graph Neural Networks: Integrating Government Ratings and Crowdsourced Reports](https://mlsanity.com/paper/2506.08740/) (2025)

## Abstract (arXiv)

Algorithms provide powerful tools for detecting and dissecting human bias and error. Here, we develop machine learning methods to to analyze how humans err in a particular high-stakes task: image interpretation. We leverage a unique dataset of 16,135,392 human predictions of whether a neighborhood voted for Donald Trump or Joe Biden in the 2020 US election, based on a Google Street View image. We show that by training a machine learning estimator of the Bayes optimal decision for each image, we can provide an actionable decomposition of human error into bias, variance, and noise terms, and further identify specific features (like pickup trucks) which lead humans astray. Our methods can be applied to ensure that human-in-the-loop decision-making is accurate and fair and are also applicable to black-box algorithmic systems.