# Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Bias

arXiv: https://arxiv.org/abs/1909.09285 · HTML: https://mlsanity.com/paper/1909.09285/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2019.

## Built on by (most cited first, 11 of 11)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions](https://mlsanity.com/paper/2404.11023/) (2024)
- [Epistemic Uncertainty-Weighted Loss for Visual Bias Mitigation](https://mlsanity.com/paper/2204.09389/) (2022)
- [How inter-rater variability relates to aleatoric and epistemic uncertainty: a case study with deep learning-based paraspinal muscle segmentation](https://mlsanity.com/paper/2308.06964/) (2023)
- [Towards Interpretability in Audio and Visual Affective Machine Learning: A Review](https://mlsanity.com/paper/2306.08933/) (2023)
- [Noise Correction on Subjective Datasets](https://mlsanity.com/paper/2311.00619/) (2023)
- [h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective](https://mlsanity.com/paper/2506.17968/) (2025)
- [Implicit Visual Bias Mitigation by Posterior Estimate Sharpening of a Bayesian Neural Network](https://mlsanity.com/paper/2303.16564/) (2023)
- [Confidence Estimation in Automatic Short Answer Grading with LLMs](https://mlsanity.com/paper/2605.00200/) (2026)

## Abstract (arXiv)

Supporting model interpretability for complex phenomena where annotators can legitimately disagree, such as emotion recognition, is a challenging machine learning task. In this work, we show that explicitly quantifying the uncertainty in such settings has interpretability benefits. We use a simple modification of a classical network inference using Monte Carlo dropout to give measures of epistemic and aleatoric uncertainty. We identify a significant correlation between aleatoric uncertainty and human annotator disagreement (r≈.3). Additionally, we demonstrate how difficult and subjective training samples can be identified using aleatoric uncertainty and how epistemic uncertainty can reveal data bias that could result in unfair predictions. We identify the total uncertainty as a suitable surrogate for model calibration, i.e. the degree we can trust model's predicted confidence. In addition to explainability benefits, we observe modest performance boosts from incorporating model uncertainty.