# Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion

arXiv: https://arxiv.org/abs/1902.03680 · HTML: https://mlsanity.com/paper/1902.03680/

## Key facts

- Cited by 127 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2019.

## Built on by (most cited first, 30 of 127)

- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [Co-learning: Learning from Noisy Labels with Self-supervision](https://mlsanity.com/paper/2108.04063/) (2021)
- [Sample Selection for Fair and Robust Training](https://mlsanity.com/paper/2110.14222/) (2021)
- [Active label cleaning for improved dataset quality under resource constraints](https://mlsanity.com/paper/2109.00574/) (2021)
- [Twin Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2303.06930/) (2023)
- [Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval](https://mlsanity.com/paper/2203.14817/) (2022)
- [Modality specific U-Net variants for biomedical image segmentation: A survey](https://mlsanity.com/paper/2107.04537/) (2021)
- [Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection](https://mlsanity.com/paper/2111.14932/) (2021)
- [Tyche: Stochastic In-Context Learning for Medical Image Segmentation](https://mlsanity.com/paper/2401.13650/) (2024)
- [SILT: Shadow-aware Iterative Label Tuning for Learning to Detect Shadows from Noisy Labels](https://mlsanity.com/paper/2308.12064/) (2023)
- [Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation](https://mlsanity.com/paper/2307.11397/) (2023)
- [Scalable Penalized Regression for Noise Detection in Learning with Noisy Labels](https://mlsanity.com/paper/2203.07788/) (2022)
- [Cross-Modal Causal Intervention for Medical Report Generation](https://mlsanity.com/paper/2303.09117/) (2023)
- [Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation](https://mlsanity.com/paper/2103.00528/) (2021)
- [Learning from Multiple Expert Annotators for Enhancing Anomaly Detection in Medical Image Analysis](https://mlsanity.com/paper/2203.10611/) (2022)
- [FedDiv: Collaborative Noise Filtering for Federated Learning with Noisy Labels](https://mlsanity.com/paper/2312.12263/) (2023)
- [Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP](https://mlsanity.com/paper/2601.09065/) (2026)
- [Learning to Rectify for Robust Learning with Noisy Labels](https://mlsanity.com/paper/2111.04239/) (2021)
- [Modeling Annotator Preference and Stochastic Annotation Error for Medical Image Segmentation](https://mlsanity.com/paper/2111.13410/) (2021)
- [Robust Node Classification on Graphs: Jointly from Bayesian Label Transition and Topology-based Label Propagation](https://mlsanity.com/paper/2208.09779/) (2022)
- [Deep Learning From Crowdsourced Labels: Coupled Cross-entropy Minimization, Identifiability, and Regularization](https://mlsanity.com/paper/2306.03288/) (2023)
- [Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls](https://mlsanity.com/paper/2208.03295/) (2022)
- [Weakly Supervised AUC Optimization: A Unified Partial AUC Approach](https://mlsanity.com/paper/2305.14258/) (2023)
- [Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowds](https://mlsanity.com/paper/2306.03116/) (2023)
- [Clean or Annotate: How to Spend a Limited Data Collection Budget](https://mlsanity.com/paper/2110.08355/) (2021)
- [Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusion](https://mlsanity.com/paper/2207.11327/) (2022)
- [Diversified and Personalized Multi-rater Medical Image Segmentation](https://mlsanity.com/paper/2403.13417/) (2024)
- [Impact of individual rater style on deep learning uncertainty in medical imaging segmentation](https://mlsanity.com/paper/2105.02197/) (2021)
- [Sample Prior Guided Robust Model Learning to Suppress Noisy Labels](https://mlsanity.com/paper/2112.01197/) (2021)

## Abstract (arXiv)

The predictive performance of supervised learning algorithms depends on the quality of labels. In a typical label collection process, multiple annotators provide subjective noisy estimates of the "truth" under the influence of their varying skill-levels and biases. Blindly treating these noisy labels as the ground truth limits the accuracy of learning algorithms in the presence of strong disagreement. This problem is critical for applications in domains such as medical imaging where both the annotation cost and inter-observer variability are high. In this work, we present a method for simultaneously learning the individual annotator model and the underlying true label distribution, using only noisy observations. Each annotator is modeled by a confusion matrix that is jointly estimated along with the classifier predictions. We propose to add a regularization term to the loss function that encourages convergence to the true annotator confusion matrix. We provide a theoretical argument as to how the regularization is essential to our approach both for the case of single annotator and multiple annotators. Despite the simplicity of the idea, experiments on image classification tasks with both simulated and real labels show that our method either outperforms or performs on par with the state-of-the-art methods and is capable of estimating the skills of annotators even with a single label available per image.