# Crowd disagreement about medical images is informative

arXiv: https://arxiv.org/abs/1806.08174 · HTML: https://mlsanity.com/paper/1806.08174/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2018.

## Built on by (most cited first, 13 of 13)

- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge](https://mlsanity.com/paper/2402.13605/) (2024)
- [Co-Correcting: Noise-tolerant Medical Image Classification via mutual Label Correction](https://mlsanity.com/paper/2109.05159/) (2021)
- [Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget](https://mlsanity.com/paper/2402.02249/) (2024)
- [Label-noise-tolerant medical image classification via self-attention and self-supervised learning](https://mlsanity.com/paper/2306.09718/) (2023)
- [Confidence Contours: Uncertainty-Aware Annotation for Medical Semantic Segmentation](https://mlsanity.com/paper/2308.07528/) (2023)
- [ENHANCE (ENriching Health data by ANnotations of Crowd and Experts): A case study for skin lesion classification](https://mlsanity.com/paper/2107.12734/) (2021)
- [Calibrating Histopathology Image Classifiers using Label Smoothing](https://mlsanity.com/paper/2201.11866/) (2022)
- [Noise Correction on Subjective Datasets](https://mlsanity.com/paper/2311.00619/) (2023)
- [What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation?](https://mlsanity.com/paper/2508.09381/) (2025)
- [Decomposing Physician Disagreement in HealthBench](https://mlsanity.com/paper/2602.22758/) (2026)

## Abstract (arXiv)

Classifiers for medical image analysis are often trained with a single consensus label, based on combining labels given by experts or crowds. However, disagreement between annotators may be informative, and thus removing it may not be the best strategy. As a proof of concept, we predict whether a skin lesion from the ISIC 2017 dataset is a melanoma or not, based on crowd annotations of visual characteristics of that lesion. We compare using the mean annotations, illustrating consensus, to standard deviations and other distribution moments, illustrating disagreement. We show that the mean annotations perform best, but that the disagreement measures are still informative. We also make the crowd annotations used in this paper available at urlhttps://figshare.com/s/5cbbce14647b66286544.