# "Is a picture of a bird a bird": Policy recommendations for dealing with ambiguity in machine vision models

arXiv: https://arxiv.org/abs/2306.15777 · HTML: https://mlsanity.com/paper/2306.15777/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 13 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 3 of 3)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Comprehensive Comparative Study of Multi-Label Classification Methods](https://mlsanity.com/paper/2102.07113/) (2021)

## Built on by (most cited first, 5 of 5)

- [The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels](https://mlsanity.com/paper/2405.05860/) (2024)
- [D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation](https://mlsanity.com/paper/2404.10857/) (2024)
- [The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models](https://mlsanity.com/paper/2310.02457/) (2023)
- [First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models](https://mlsanity.com/paper/2311.05020/) (2023)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)

## Abstract (arXiv)

Many questions that we ask about the world do not have a single clear answer, yet typical human annotation set-ups in machine learning assume there must be a single ground truth label for all examples in every task. The divergence between reality and practice is stark, especially in cases with inherent ambiguity and where the range of different subjective judgments is wide. Here, we examine the implications of subjective human judgments in the behavioral task of labeling images used to train machine vision models. We identify three primary sources of ambiguity arising from (i) depictions of labels in the images, (ii) raters' backgrounds, and (iii) the task definition. On the basis of the empirical results, we suggest best practices for handling label ambiguity in machine learning datasets.