mlsanityData to 7 Oct 2026

On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD

arXiv 2603.10397 ↗First posted 11 Mar 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

One crucial factor behind the success of deep learning lies in the implicit bias induced by noise inherent in gradient-based training algorithms. Motivated by empirical observations that training with noisy labels improves model generalization, we delve into the underlying mechanisms behind stochastic gradient descent (SGD) with label noise. Focusing on a two-layer over-parameterized linear network, we analyze the learning dynamics of label noise SGD, unveiling a two-phase learning behavior. In Phase I, the magnitudes of model weights progressively diminish, and the model escapes the lazy regime; enters the rich regime. In Phase II, the alignment between model weights and the ground-truth interpolator increases, and the model eventually converges. Our analysis highlights the critical role of label noise in driving the transition from the lazy to the rich regime and minimally explains its empirical success. Furthermore, we extend these insights to Sharpness-Aware Minimization (SAM), showing that the principles governing label noise SGD also apply to broader optimization algorithms. Extensive experiments, conducted under both synthetic and real-world setups, strongly support our theory. Our code is released at https://github.com/a-usually/Label-Noise-SGD.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 52 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

46 of 52 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Wide Residual Networkscited by 3,057
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  8. 2018On Lazy Training in Differentiable Programmingcited by 671
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  11. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
34 more
  1. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  2. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  3. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  4. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  5. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  6. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  7. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  8. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  9. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  10. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  11. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  12. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  13. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  14. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  15. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  16. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  17. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  18. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  19. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  20. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  21. 2020On the training dynamics of deep networks with L2 regularizationcited by 38
  22. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  23. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  24. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  25. 2023Pruning Deep Neural Networks from a Sparsity Perspectivecited by 25
  26. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  27. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  28. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  29. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  30. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  31. 2024Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspectivecited by 12
  32. 2025On the Role of Label Noise in the Feature Learning Processcited by 4
  33. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  34. 2023Generalization Bounds for Label Noise Stochastic Gradient Descentcited by 3

Built on by

Not cited yet.