mlsanityData to 7 Oct 2026

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

arXiv 2302.08982 ↗First posted 17 Feb 2023cs.LG math.OC stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over diagonal linear networks. We prove the convergence of GD and SGD with macroscopic stepsizes in an overparametrised regression setting and characterise their solutions through an implicit regularisation problem. Our crisp characterisation leads to qualitative insights about the impact of stochasticity and stepsizes on the recovered solution. Specifically, we show that large stepsizes consistently benefit SGD for sparse regression problems, while they can hinder the recovery of sparse solutions for GD. These effects are magnified for stepsizes in a tight window just below the divergence threshold, in the "edge of stability" regime. Our findings are supported by experimental results.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 47 of its 69 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

47 of 69 references.

References · 47

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  9. 2017Implicit Regularization in Matrix Factorizationcited by 318
  10. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  11. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  12. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
35 more
  1. 2017Three Factors Influencing Minima in SGDcited by 270
  2. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  3. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  4. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  5. 2020Directional convergence and alignment in deep learningcited by 172
  6. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  7. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  8. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  9. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  10. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  11. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  12. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  13. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  14. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 100
  15. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  16. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  17. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  18. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  19. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  20. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  21. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 56
  22. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  23. 2022Learning threshold neurons via the "edge of stability"cited by 50
  24. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  25. 2013Margins, Shrinkage, and Boostingcited by 35
  26. 2019Exponentiated Gradient Meets Gradient Descentcited by 32
  27. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  28. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  29. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  30. 2020The Statistical Complexity of Early-Stopped Mirror Descentcited by 19
  31. 2021Fast Stochastic Bregman Gradient Methods: Sharp Analysis and Variance Reductioncited by 19
  32. 2013A Generalized Online Mirror Descent with Applications to Classification and Regressioncited by 18
  33. 2020A Continuous-Time Mirror Descent Approach to Sparse Phase Retrievalcited by 9
  34. 2022On the Benefits of Large Learning Rates for Kernel Methodscited by 8
  35. 2021Concentration of Non-Isotropic Random Tensors with Applications to Learning and Empirical Risk Minimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 6; 2026: 17.

35 papers.

Sort:

By area:

  1. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 31
  2. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  3. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 16
  4. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  5. 2024Plastic Learning with Deep Fourier Featurescited by 14
  6. 2023Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrizationcited by 11
  7. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 10
  8. 2024Implicit Bias of Mirror Flow on Separable Datacited by 7
  9. 2024Swing-by Dynamics in Concept Learning and Compositional Generalizationcited by 5
  10. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 5
  11. 2023Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networkscited by 4
  12. 2024Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networkscited by 3
23 more
  1. 2026Understanding and inverse design of implicit bias in stochastic learning: a geometric perspectivecited by 3
  2. 2026Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 3
  3. 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
  4. 2024Optimization Insights into Deep Diagonal Linear Networkscited by 2
  5. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  6. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  7. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 2
  8. 2024The Implicit Bias of Heterogeneity towards Invariance: A Study of Multi-Environment Matrix Sensingcited by 1
  9. 2025Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theorycited by 1
  10. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  11. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  12. 2024Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regimecited by 0
  13. 2026Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalizationcited by 0
  14. 2026On the implicit regularization of Langevin dynamics with projected noisecited by 0
  15. 2026Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low Dimensioncited by 0
  16. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 0
  17. 2026Hard labels sampled from sparse targets mislead rotation invariant algorithmscited by 0
  18. 2026A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Modelscited by 0
  19. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0
  20. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  21. 2026Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathwaycited by 0
  22. 2026Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initializationcited by 0
  23. 2026How Edge of Stability Hinders SCAFFOLD in Federated Optimizationcited by 0