mlsanityData to 14 Oct 2026

On the Role of Label Noise in the Feature Learning Process

arXiv 2505.18909 ↗First posted 25 May 2025stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Deep learning with noisy labels presents significant challenges. In this work, we theoretically characterize the role of label noise from a feature learning perspective. Specifically, we consider a signal-noise data distribution, where each sample comprises a label-dependent signal and label-independent noise, and rigorously analyze the training dynamics of a two-layer convolutional neural network under this data setup, along with the presence of label noise. Our analysis identifies two key stages. In Stage I, the model perfectly fits all the clean samples (i.e., samples without label noise) while ignoring the noisy ones (i.e., samples with noisy labels). During this stage, the model learns the signal from the clean samples, which generalizes well on unseen data. In Stage II, as the training loss converges, the gradient in the direction of noise surpasses that of the signal, leading to overfitting on noisy samples. Eventually, the model memorizes the noise present in the noisy samples and degrades its generalization ability. Furthermore, our analysis provides a theoretical basis for two widely used techniques for tackling label noise: early stopping and sample selection. Experiments on both synthetic and real-world setups validate our theory.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 50 of its 71 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

50 of 71 references.

References · 50

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  5. 2017A Closer Look at Memorization in Deep Networkscited by 942
  6. 2018On Lazy Training in Differentiable Programmingcited by 675
  7. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 416
  10. 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
  11. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
38 more
  1. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  2. 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
  3. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  4. 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
  5. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  6. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  7. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  8. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  9. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  10. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  11. 2021Understanding and Improving Early Stopping for Learning with Noisy Labelscited by 113
  12. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  13. 2022Vision Transformers provably learn spatial structurecited by 93
  14. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  15. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  16. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  17. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  18. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  19. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  20. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  21. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  22. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
  23. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 28
  24. 2023Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labelscited by 20
  25. 2024On the Feature Learning in Diffusion Modelscited by 20
  26. 2023The Benefits of Mixup for Feature Learningcited by 19
  27. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  28. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  29. 2024Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalizationcited by 17
  30. 2025Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?cited by 15
  31. 2022Provably Learning Diverse Features in Multi-View Data with Midpoint Mixupcited by 14
  32. 2020When does gradient descent with logistic loss find interpolating two-layer networks?cited by 12
  33. 2023Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptronscited by 12
  34. 2023Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Datacited by 10
  35. 2024On the Comparison between Multi-modal and Single-modal Contrastive Learningcited by 10
  36. 2023Regularly Truncated M-estimators for Learning with Noisy Labelscited by 8
  37. 2024Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learningcited by 6
  38. 2023On Emergence of Clean-Priority Learning in Early Stopped Neural Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  2. 2026Linear Dynamics in the RLVR Training of Large Language Modelscited by 4
  3. 2026Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspectivecited by 1
  4. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0