# Robustness and Reliability When Training With Noisy Labels

arXiv: https://arxiv.org/abs/2110.03321 · HTML: https://mlsanity.com/paper/2110.03321/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks](https://mlsanity.com/paper/1903.11680/) (2019)
- [Making Risk Minimization Tolerant to Label Noise](https://mlsanity.com/paper/1403.3610/) (2014)
- [On Symmetric Losses for Learning from Corrupted Labels](https://mlsanity.com/paper/1901.09314/) (2019)

## Built on by (most cited first, 10 of 10)

- [A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?](https://mlsanity.com/paper/2404.01775/) (2024)
- [Robustness and Cybersecurity in the EU Artificial Intelligence Act](https://mlsanity.com/paper/2502.16184/) (2025)
- [Collective Human Opinions in Semantic Textual Similarity](https://mlsanity.com/paper/2308.04114/) (2023)
- [Regularly Truncated M-estimators for Learning with Noisy Labels](https://mlsanity.com/paper/2309.00894/) (2023)
- [Noisy Annotations in Semantic Segmentation](https://mlsanity.com/paper/2406.10891/) (2024)
- [A Survey on Preserving Fairness Guarantees in Changing Environments](https://mlsanity.com/paper/2211.07530/) (2022)
- [Synergy-Guided Regional Supervision of Pseudo Labels for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2411.04493/) (2024)
- [PASS: Peer-Agreement based Sample Selection for training with Noisy Labels](https://mlsanity.com/paper/2303.10802/) (2023)
- [Bounded Adjustment with Reliability-Guided Embedding for Imbalanced Learning with Noisy Labels](https://mlsanity.com/paper/2609.16380/) (2026)
- [Noisy-Label Calibration with Multiple Unreliable Labels](https://mlsanity.com/paper/2610.19543/) (2026)

## Abstract (arXiv)

Labelling of data for supervised learning can be costly and time-consuming and the risk of incorporating label noise in large data sets is imminent. When training a flexible discriminative model using a strictly proper loss, such noise will inevitably shift the solution towards the conditional distribution over noisy labels. Nevertheless, while deep neural networks have proven capable of fitting random labels, regularisation and the use of robust loss functions empirically mitigate the effects of label noise. However, such observations concern robustness in accuracy, which is insufficient if reliable uncertainty quantification is critical. We demonstrate this by analysing the properties of the conditional distribution over noisy labels for an input-dependent noise model. In addition, we evaluate the set of robust loss functions characterised by noise-insensitive, asymptotic risk minimisers. We find that strictly proper and robust loss functions both offer asymptotic robustness in accuracy, but neither guarantee that the final model is calibrated. Moreover, even with robust loss functions, overfitting is an issue in practice. With these results, we aim to explain observed robustness of common training practices, such as early stopping, to label noise. In addition, we aim to encourage the development of new noise-robust algorithms that not only preserve accuracy but that also ensure reliability.