# Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion

arXiv: https://arxiv.org/abs/2106.09291 · HTML: https://mlsanity.com/paper/2106.09291/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 30 of 30)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Decoupling "when to update" from "how to update"](https://mlsanity.com/paper/1706.02613/) (2017)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [Probabilistic End-to-end Noise Correction for Learning with Noisy Labels](https://mlsanity.com/paper/1903.07788/) (2019)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Are Anchor Points Really Indispensable in Label-Noise Learning?](https://mlsanity.com/paper/1906.00189/) (2019)
- [Dimensionality-Driven Learning with Noisy Labels](https://mlsanity.com/paper/1806.02612/) (2018)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks](https://mlsanity.com/paper/1903.11680/) (2019)
- [Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey](https://mlsanity.com/paper/1912.05170/) (2019)
- [Masking: A New Perspective of Noisy Supervision](https://mlsanity.com/paper/1805.08193/) (2018)
- [Part-dependent Label Noise: Towards Instance-dependent Label Noise](https://mlsanity.com/paper/2006.07836/) (2020)
- [L_DMI: An Information-theoretic Noise-robust Loss Function](https://mlsanity.com/paper/1909.03388/) (2019)
- [SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning](https://mlsanity.com/paper/2012.00925/) (2020)

## Built on by (most cited first, 30 of 36)

- [ProMix: Combating Label Noise via Maximizing Clean Sample Utility](https://mlsanity.com/paper/2207.10276/) (2022)
- [Rank-based Decomposable Losses in Machine Learning: A Survey](https://mlsanity.com/paper/2207.08768/) (2022)
- [SELC: Self-Ensemble Label Correction Improves Learning with Noisy Labels](https://mlsanity.com/paper/2205.01156/) (2022)
- [Mitigating Label Noise through Data Ambiguation](https://mlsanity.com/paper/2305.13764/) (2023)
- [Learning Discriminative Dynamics with Label Corruption for Noisy Label Detection](https://mlsanity.com/paper/2405.19902/) (2024)
- [Foster Adaptivity and Balance in Learning with Noisy Labels](https://mlsanity.com/paper/2407.02778/) (2024)
- [Regroup Median Loss for Combating Label Noise](https://mlsanity.com/paper/2312.06273/) (2023)
- [Noisy Label Processing for Classification: A Survey](https://mlsanity.com/paper/2404.04159/) (2024)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [Learning with Imbalanced Noisy Data by Preventing Bias in Sample Selection](https://mlsanity.com/paper/2402.11242/) (2024)
- [A Survey of Dataset Refinement for Problems in Computer Vision Datasets](https://mlsanity.com/paper/2210.11717/) (2022)
- [An Analysis of Concept Bottleneck Models: Measuring, Understanding, and Mitigating the Impact of Noisy Annotations](https://mlsanity.com/paper/2505.16705/) (2025)
- [Learning with Noisy Labels through Learnable Weighting and Centroid Similarity](https://mlsanity.com/paper/2303.09470/) (2023)
- [Local or Global: Selective Knowledge Assimilation for Federated Learning with Limited Labels](https://mlsanity.com/paper/2307.08809/) (2023)
- [Noisy Node Classification by Bi-level Optimization based Multi-teacher Distillation](https://mlsanity.com/paper/2404.17875/) (2024)
- [Optimized Gradient Clipping for Noisy Label Learning](https://mlsanity.com/paper/2412.08941/) (2024)
- [Learning from Long-Tailed Noisy Data with Sample Selection and Balanced Loss](https://mlsanity.com/paper/2211.10906/) (2022)
- [Efficient Adaptive Label Refinement for Label Noise Learning](https://mlsanity.com/paper/2502.00386/) (2025)
- [Robust Learning of Diffusion Models with Extremely Noisy Conditions](https://mlsanity.com/paper/2510.10149/) (2025)
- [Can LLMs Learn to Reason Robustly under Noisy Supervision?](https://mlsanity.com/paper/2604.03993/) (2026)
- [Towards the Identifiability in Noisy Label Learning: A Multinomial Mixture Modelling Approach](https://mlsanity.com/paper/2301.01405/) (2023)
- [Robust Noisy Label Learning via Two-Stream Sample Distillation](https://mlsanity.com/paper/2404.10499/) (2024)
- [Identifying and Correcting Label Noise for Robust GNNs via Influence Contradiction](https://mlsanity.com/paper/2601.17469/) (2026)
- [Contrastive Representations for Label Noise Require Fine-Tuning](https://mlsanity.com/paper/2108.09154/) (2021)
- [Pairwise Similarity Distribution Clustering for Noisy Label Learning](https://mlsanity.com/paper/2404.01853/) (2024)
- [Energy Guided smoothness to improve Robustness in Graph Classification](https://mlsanity.com/paper/2412.08419/) (2024)
- [LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training](https://mlsanity.com/paper/2509.20786/) (2025)
- [Revisiting Meta-Learning with Noisy Labels: Reweighting Dynamics and Theoretical Guarantees](https://mlsanity.com/paper/2510.12209/) (2025)
- [DREAM: Dual-Standard Semantic Homogeneity with Dynamic Optimization for Graph Learning with Label Noise](https://mlsanity.com/paper/2601.17449/) (2026)
- [FedRG: Unleashing the Representation Geometry for Federated Learning with Noisy Clients](https://mlsanity.com/paper/2603.19722/) (2026)

## Abstract (arXiv)

Deep neural networks need large amounts of labeled data to achieve good performance. In real-world applications, labels are usually collected from non-experts such as crowdsourcing to save cost and thus are noisy. In the past few years, deep learning methods for dealing with noisy labels have been developed, many of which are based on the small-loss criterion. However, there are few theoretical analyses to explain why these methods could learn well from noisy labels. In this paper, we theoretically explain why the widely-used small-loss criterion works. Based on the explanation, we reformalize the vanilla small-loss criterion to better tackle noisy labels. The experimental results verify our theoretical explanation and also demonstrate the effectiveness of the reformalization.