# Understanding and Improving Early Stopping for Learning with Noisy Labels

arXiv: https://arxiv.org/abs/2106.15853 · HTML: https://mlsanity.com/paper/2106.15853/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [Are Anchor Points Really Indispensable in Label-Noise Learning?](https://mlsanity.com/paper/1906.00189/) (2019)
- [Dimensionality-Driven Learning with Noisy Labels](https://mlsanity.com/paper/1806.02612/) (2018)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)
- [Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning](https://mlsanity.com/paper/2006.07805/) (2020)
- [Iterative Learning with Open-set Noisy Labels](https://mlsanity.com/paper/1804.00092/) (2018)
- [Curriculum Loss: Robust Learning and Generalization against Label Corruption](https://mlsanity.com/paper/1905.10045/) (2019)
- [Do We Need Zero Training Loss After Achieving Zero Training Error?](https://mlsanity.com/paper/2002.08709/) (2020)
- [Part-dependent Label Noise: Towards Instance-dependent Label Noise](https://mlsanity.com/paper/2006.07836/) (2020)
- [Class2Simi: A Noise Reduction Perspective on Learning with Noisy Labels](https://mlsanity.com/paper/2006.07831/) (2020)

## Built on by (most cited first, 30 of 113)

- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [Label-free Node Classification on Graphs with Large Language Models (LLMS)](https://mlsanity.com/paper/2310.04668/) (2023)
- [Detecting Corrupted Labels Without Training a Model to Predict](https://mlsanity.com/paper/2110.06283/) (2021)
- [Instance-Dependent Label-Noise Learning with Manifold-Regularized Transition Matrix Estimation](https://mlsanity.com/paper/2206.02791/) (2022)
- [ProMix: Combating Label Noise via Maximizing Clean Sample Utility](https://mlsanity.com/paper/2207.10276/) (2022)
- [Mitigating Memorization of Noisy Labels by Clipping the Model Prediction](https://mlsanity.com/paper/2212.04055/) (2022)
- [Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalization](https://mlsanity.com/paper/2208.11351/) (2022)
- [The Rich Get Richer: Disparate Impact of Semi-Supervised Learning](https://mlsanity.com/paper/2110.06282/) (2021)
- [From Shortcuts to Triggers: Backdoor Defense with Denoised PoE](https://mlsanity.com/paper/2305.14910/) (2023)
- [Early Stopping Against Label Noise Without Validation Data](https://mlsanity.com/paper/2502.07551/) (2025)
- [On the Over-Memorization During Natural, Robust and Catastrophic Overfitting](https://mlsanity.com/paper/2310.08847/) (2023)
- [FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning](https://mlsanity.com/paper/2310.16412/) (2023)
- [DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation](https://mlsanity.com/paper/2411.19946/) (2024)
- [Late Stopping: Avoiding Confidently Learning from Mislabeled Examples](https://mlsanity.com/paper/2308.13862/) (2023)
- [FedA3I: Annotation Quality-Aware Aggregation for Federated Medical Image Segmentation against Heterogeneous Annotation Noise](https://mlsanity.com/paper/2312.12838/) (2023)
- [Two Decades of Bengali Handwritten Digit Recognition: A Survey](https://mlsanity.com/paper/2206.02234/) (2022)
- [Labeling Chaos to Learning Harmony: Federated Learning with Noisy Labels](https://mlsanity.com/paper/2208.09378/) (2022)
- [InstanT: Semi-supervised Learning with Instance-dependent Thresholds](https://mlsanity.com/paper/2310.18910/) (2023)
- [FedNoiL: A Simple Two-Level Sampling Method for Federated Learning with Noisy Labels](https://mlsanity.com/paper/2205.10110/) (2022)
- [Toward Robustness in Multi-label Classification: A Data Augmentation Strategy against Imbalance and Noise](https://mlsanity.com/paper/2312.07087/) (2023)
- [PADDLES: Phase-Amplitude Spectrum Disentangled Early Stopping for Learning with Noisy Labels](https://mlsanity.com/paper/2212.03462/) (2022)
- [When does Privileged Information Explain Away Label Noise?](https://mlsanity.com/paper/2303.01806/) (2023)
- [Blind Knowledge Distillation for Robust Image Classification](https://mlsanity.com/paper/2211.11355/) (2022)
- [Latent Class-Conditional Noise Model](https://mlsanity.com/paper/2302.09595/) (2023)
- [Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations](https://mlsanity.com/paper/2305.12715/) (2023)
- [Foster Adaptivity and Balance in Learning with Noisy Labels](https://mlsanity.com/paper/2407.02778/) (2024)
- [Regroup Median Loss for Combating Label Noise](https://mlsanity.com/paper/2312.06273/) (2023)
- [Noisy Label Processing for Classification: A Survey](https://mlsanity.com/paper/2404.04159/) (2024)
- [Bootstrapping the Relationship Between Images and Their Clean and Noisy Labels](https://mlsanity.com/paper/2210.08826/) (2022)
- [A Layer Selection Approach to Test Time Adaptation](https://mlsanity.com/paper/2404.03784/) (2024)

## Abstract (arXiv)

The memorization effect of deep neural network (DNN) plays a pivotal role in many state-of-the-art label-noise learning methods. To exploit this property, the early stopping trick, which stops the optimization at the early stage of training, is usually adopted. Current methods generally decide the early stopping point by considering a DNN as a whole. However, a DNN can be considered as a composition of a series of layers, and we find that the latter layers in a DNN are much more sensitive to label noise, while their former counterparts are quite robust. Therefore, selecting a stopping point for the whole network may make different DNN layers antagonistically affected each other, thus degrading the final performance. In this paper, we propose to separate a DNN into different parts and progressively train them to address this problem. Instead of the early stopping, which trains a whole DNN all at once, we initially train former DNN layers by optimizing the DNN with a relatively large number of epochs. During training, we progressively train the latter DNN layers by using a smaller number of epochs with the preceding layers fixed to counteract the impact of noisy labels. We term the proposed method as progressive early stopping (PES). Despite its simplicity, compared with the early stopping, PES can help to obtain more promising and stable results. Furthermore, by combining PES with existing approaches on noisy label training, we achieve state-of-the-art performance on image classification benchmarks.