mlsanityData to 7 Oct 2026

Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisation

arXiv 2206.09841 ↗First posted 20 Jun 2022stat.ML cs.LG math.OCcited by 29

Not on the map. Why?

Abstract (arXiv)

Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the role of the label noise in the training dynamics of a quadratically parametrised model through its continuous time version. We explicitly characterise the solution chosen by the stochastic flow and prove that it implicitly solves a Lasso program. To fully complete our analysis, we provide nonasymptotic convergence guarantees for the dynamics as well as conditions for support recovery. We also give experimental results which support our theoretical claims. Our findings highlight the fact that structured noise can induce better generalisation and help explain the greater performances of stochastic dynamics as observed in practice.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 20 Jun 2022.

Builds on

25 of 37 references.

References · 25

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  8. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  11. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  12. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
13 more
  1. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  2. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  3. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  4. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  5. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  6. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  7. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  8. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  9. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
  10. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  11. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  12. 2019Exponentiated Gradient Meets Gradient Descentcited by 32
  13. 2020A Continuous-Time Mirror Descent Approach to Sparse Phase Retrievalcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 7; 2024: 7; 2025: 5; 2026: 5.

29 papers.

Sort:

  1. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  2. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  3. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  4. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  5. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  6. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  7. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  8. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  9. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  10. 2024Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspectivecited by 12
  11. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 11
  12. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
17 more
  1. 2022Flatter, faster: scaling momentum for optimal speedup of SGDcited by 5
  2. 2024Simplicity Bias via Global Convergence of Sharpness Minimizationcited by 4
  3. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  4. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  5. 2023Generalization Bounds for Label Noise Stochastic Gradient Descentcited by 3
  6. 2024Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descentcited by 3
  7. 2025Diagonal Linear Networks and the Lasso Regularization Pathcited by 3
  8. 2026Understanding and inverse design of implicit bias in stochastic learning: a geometric perspectivecited by 3
  9. 2022Noise Injection as a Probe of Deep Learning Dynamicscited by 2
  10. 2024Improving Adaptivity via Over-Parameterization in Sequence Modelscited by 2
  11. 2024A Mirror Descent Perspective of Smoothed Sign Descentcited by 2
  12. 2024The Implicit Bias of Heterogeneity towards Invariance: A Study of Multi-Environment Matrix Sensingcited by 1
  13. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  14. 2025Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysiscited by 0
  15. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0
  16. 2026Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Ratecited by 0
  17. 2026Langevin dynamics along the zero set of real-analytic potentialscited by 0