# On the Ramifications of Human Label Uncertainty

arXiv: https://arxiv.org/abs/2211.05871 · HTML: https://mlsanity.com/paper/2211.05871/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 10 Nov 2022.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images](https://mlsanity.com/paper/1808.01097/) (2018)
- [To Aggregate or Not? Learning with Separate Noisy Labels](https://mlsanity.com/paper/2206.07181/) (2022)
- [OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics](https://mlsanity.com/paper/2209.11195/) (2022)
- [Introspective Learning : A Two-Stage Approach for Inference in Neural Networks](https://mlsanity.com/paper/2209.08425/) (2022)
- [Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation](https://mlsanity.com/paper/2103.00528/) (2021)
- [UNIQUE: Unsupervised Image Quality Estimation](https://mlsanity.com/paper/1810.06631/) (2018)
- [Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation](https://mlsanity.com/paper/2011.02696/) (2020)
- [MS-UNIQUE: Multi-model and Sharpness-weighted Unsupervised Image Quality Estimation](https://mlsanity.com/paper/1811.08947/) (2018)

## Built on by (most cited first, 3 of 3)

- [VOICE: Variance of Induced Contrastive Explanations to quantify Uncertainty in Neural Network Interpretability](https://mlsanity.com/paper/2406.00573/) (2024)
- [Counterfactual Gradients-based Quantification of Prediction Trust in Neural Networks](https://mlsanity.com/paper/2405.13758/) (2024)
- [Structured Credal Learning](https://mlsanity.com/paper/2603.14070/) (2026)

## Abstract (arXiv)

Humans exhibit disagreement during data labeling. We term this disagreement as human label uncertainty. In this work, we study the ramifications of human label uncertainty (HLU). Our evaluation of existing uncertainty estimation algorithms, with the presence of HLU, indicates the limitations of existing uncertainty metrics and algorithms themselves in response to HLU. Meanwhile, we observe undue effects in predictive uncertainty and generalizability. To mitigate the undue effects, we introduce a novel natural scene statistics (NSS) based label dilution training scheme without requiring massive human labels. Specifically, we first select a subset of samples with low perceptual quality ranked by statistical regularities of images. We then assign separate labels to each sample in this subset to obtain a training set with diluted labels. Our experiments and analysis demonstrate that training with NSS-based label dilution alleviates the undue effects caused by HLU.