mlsanityData to 7 Oct 2026

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

arXiv 2106.09524 ↗First posted 17 Jun 2021cs.LGcited by 107

Not on the map. Why?

Abstract (arXiv)

Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the dynamics of stochastic gradient descent over diagonal linear networks through its continuous time version, namely stochastic gradient flow. We explicitly characterise the solution chosen by the stochastic flow and prove that it always enjoys better generalisation properties than that of gradient flow. Quite surprisingly, we show that the convergence speed of the training loss controls the magnitude of the biasing effect: the slower the convergence, the better the bias. To fully complete our analysis, we provide convergence guarantees for the dynamics. We also give experimental results which support our theoretical claims. Our findings highlight the fact that structured noise can induce better generalisation and they help explain the greater performances observed in practice of stochastic gradient descent over gradient descent.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

29 of 41 references.

References · 29

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  11. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  12. 2017Three Factors Influencing Minima in SGDcited by 270
17 more
  1. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  2. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  3. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  4. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  5. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  6. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  7. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  8. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  9. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  10. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  11. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
  12. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 56
  13. 2021Last iterate convergence of SGD for Least-Squares in the Interpolation regimecited by 41
  14. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  15. 2019Exponentiated Gradient Meets Gradient Descentcited by 32
  16. 2020The Statistical Complexity of Early-Stopped Mirror Descentcited by 19
  17. 2020A Continuous-Time Mirror Descent Approach to Sparse Phase Retrievalcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 21; 2023: 19; 2024: 30; 2025: 18; 2026: 18.

107 papers.

Sort:

By area:

  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  2. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  3. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  4. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  5. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  6. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  7. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  8. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  9. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 65
  10. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  11. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  12. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
38 more
  1. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 51
  2. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  3. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  4. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  5. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  6. 2023Deep networks for system identification: a Surveycited by 35
  7. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  8. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  9. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  10. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  11. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  12. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  13. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 25
  14. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  15. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  16. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  17. 2022Infinite-width limit of deep linear neural networkscited by 20
  18. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  19. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  20. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  21. 2022Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measurescited by 17
  22. 2022Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Biascited by 16
  23. 2024How do Transformers perform In-Context Autoregressive Learning?cited by 16
  24. 2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
  25. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  26. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  27. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 13
  28. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  29. 2024Mask in the Mirror: Implicit Sparsificationcited by 12
  30. 2022Entropic Descent Archetypal Analysis for Blind Hyperspectral Unmixingcited by 11
  31. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  32. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  33. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  34. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 10
  35. 2023Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learningcited by 9
  36. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  37. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
  38. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9

50 most cited of 107 All 107 (JSON)