mlsanityData to 13 Oct 2026

Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion

arXiv 2106.09291 ↗First posted 17 Jun 2021cs.LG stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

Deep neural networks need large amounts of labeled data to achieve good performance. In real-world applications, labels are usually collected from non-experts such as crowdsourcing to save cost and thus are noisy. In the past few years, deep learning methods for dealing with noisy labels have been developed, many of which are based on the small-loss criterion. However, there are few theoretical analyses to explain why these methods could learn well from noisy labels. In this paper, we theoretically explain why the widely-used small-loss criterion works. Based on the explanation, we reformalize the vanilla small-loss criterion to better tackle noisy labels. The experimental results verify our theoretical explanation and also demonstrate the effectiveness of the reformalization.

Key facts. Cited by 36 papers in the map (data through 13 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

30 of 45 references.

References · 30

  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
  2. 2021Convex optimizationcited by 2,488
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,666
  4. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  5. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  6. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,055
  7. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  8. 2017A Closer Look at Memorization in Deep Networkscited by 939
  9. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  10. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  11. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  12. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 415
18 more
  1. 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
  2. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
  3. 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
  4. 2020Combating noisy labels by agreement: A joint training method with co-regularizationcited by 289
  5. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  6. 2017Decoupling "when to update" from "how to update"cited by 230
  7. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  8. 2019Probabilistic End-to-end Noise Correction for Learning with Noisy Labelscited by 210
  9. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  10. 2019Are Anchor Points Really Indispensable in Label-Noise Learning?cited by 208
  11. 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
  12. 2017Deep Learning is Robust to Massive Label Noisecited by 164
  13. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  14. 2019Image Classification with Deep Learning in the Presence of Noisy Labels: A Surveycited by 101
  15. 2018Masking: A New Perspective of Noisy Supervisioncited by 93
  16. 2020Part-dependent Label Noise: Towards Instance-dependent Label Noisecited by 40
  17. 2019L_DMI: An Information-theoretic Noise-robust Loss Functioncited by 26
  18. 2020SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learningcited by 11

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 6; 2024: 9; 2025: 5; 2026: 10.

36 papers.

Sort:

By area:

  1. 2022ProMix: Combating Label Noise via Maximizing Clean Sample Utilitycited by 48
  2. 2022Rank-based Decomposable Losses in Machine Learning: A Surveycited by 26
  3. 2022SELC: Self-Ensemble Label Correction Improves Learning with Noisy Labelscited by 24
  4. 2023Mitigating Label Noise through Data Ambiguationcited by 10
  5. 2024Foster Adaptivity and Balance in Learning with Noisy Labelscited by 9
  6. 2023Regroup Median Loss for Combating Label Noisecited by 8
  7. 2024Noisy Label Processing for Classification: A Surveycited by 8
  8. 2024Learning Discriminative Dynamics with Label Corruption for Noisy Label Detectioncited by 8
  9. 2023Data Optimization in Deep Learning: A Surveycited by 6
  10. 2024Learning with Imbalanced Noisy Data by Preventing Bias in Sample Selectioncited by 6
  11. 2022A Survey of Dataset Refinement for Problems in Computer Vision Datasetscited by 5
  12. 2025An Analysis of Concept Bottleneck Models: Measuring, Understanding, and Mitigating the Impact of Noisy Annotationscited by 5
24 more
  1. 2023Learning with Noisy Labels through Learnable Weighting and Centroid Similaritycited by 4
  2. 2023Local or Global: Selective Knowledge Assimilation for Federated Learning with Limited Labelscited by 4
  3. 2024Noisy Node Classification by Bi-level Optimization based Multi-teacher Distillationcited by 4
  4. 2024Optimized Gradient Clipping for Noisy Label Learningcited by 4
  5. 2022Learning from Long-Tailed Noisy Data with Sample Selection and Balanced Losscited by 2
  6. 2025Efficient Adaptive Label Refinement for Label Noise Learningcited by 2
  7. 2025Robust Learning of Diffusion Models with Extremely Noisy Conditionscited by 2
  8. 2026Can LLMs Learn to Reason Robustly under Noisy Supervision?cited by 2
  9. 2023Towards the Identifiability in Noisy Label Learning: A Multinomial Mixture Modelling Approachcited by 1
  10. 2024Robust Noisy Label Learning via Two-Stream Sample Distillationcited by 1
  11. 2026Identifying and Correcting Label Noise for Robust GNNs via Influence Contradictioncited by 1
  12. 2021Contrastive Representations for Label Noise Require Fine-Tuningcited by 0
  13. 2024Pairwise Similarity Distribution Clustering for Noisy Label Learningcited by 0
  14. 2024Energy Guided smoothness to improve Robustness in Graph Classificationcited by 0
  15. 2025LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Trainingcited by 0
  16. 2025Revisiting Meta-Learning with Noisy Labels: Reweighting Dynamics and Theoretical Guaranteescited by 0
  17. 2026DREAM: Dual-Standard Semantic Homogeneity with Dynamic Optimization for Graph Learning with Label Noisecited by 0
  18. 2026FedRG: Unleashing the Representation Geometry for Federated Learning with Noisy Clientscited by 0
  19. 2026See Through the Noise: Improving Domain Generalization in Gaze Estimationcited by 0
  20. 2026Holistic Reliability Propagation: Decoupling Annotation and Prediction for Robust Noisy-Labelcited by 0
  21. 2026GAMR: Geometric-Aware Manifold Regularization with Virtual Outlier Synthesis for Learning with Noisy Labelscited by 0
  22. 2026Rethinking Noise-Robust Training for Frozen Vision Foundation Models: A Cross-Dataset Benchmark with a Case Study of Small-Loss Failurecited by 0
  23. 2026Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teachingcited by 0
  24. 2026PaSta: Noisy Node Classification with Partial Label Learningcited by 0