mlsanityData to 7 Oct 2026

Understanding Gradient Descent on Edge of Stability in Deep Learning

arXiv 2205.09745 ↗First posted 19 May 2022cs.LG math.OCcited by 126

Not on the map. Why?

Abstract (arXiv)

Deep learning experiments by Cohen et al. [2021] using deterministic Gradient Descent (GD) revealed an Edge of Stability (EoS) phase when learning rate (LR) and sharpness (i.e., the largest eigenvalue of Hessian) no longer behave as in traditional optimization. Sharpness stabilizes around 2/LR and loss goes up and down across iterations, yet still with an overall downward trend. The current paper mathematically analyzes a new mechanism of implicit regularization in the EoS phase, whereby GD updates due to non-smooth loss landscape turn out to evolve along some deterministic flow on the manifold of minimum loss. This is in contrast to many previous results about implicit bias either relying on infinitesimal updates or noise in gradient. Formally, for any smooth function L with certain regularity condition, this effect is demonstrated for (1) Normalized GD, i.e., GD with a varying LR ηt =fracη| nabla L(x(t)) | and loss L; (2) GD with constant LR and loss √L- minx L(x). Both provably enter the Edge of Stability, with the associated flow on the manifold minimizing λ1(nabla2 L). The above theoretical results have been corroborated by an experimental study.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 33 of its 67 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

33 of 67 references.

References · 33

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2018On Lazy Training in Differentiable Programmingcited by 671
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  12. 2017How to Escape Saddle Points Efficientlycited by 284
21 more
  1. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  2. 2017Searching for Activation Functionscited by 158
  3. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  4. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  5. 2020Implicit Gradient Regularizationcited by 145
  6. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  7. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  8. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  9. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  10. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  11. 2022Understanding the unstable convergence of gradient descentcited by 84
  12. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  13. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  14. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  15. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  16. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  17. 2020Mirrorless Mirror Descent: A Natural Derivation of Mirror Descentcited by 18
  18. 2017First-order Methods Almost Always Avoid Saddle Pointscited by 14
  19. 2019Positively Scale-Invariant Flatness of ReLU Neural Networkscited by 11
  20. 2017Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activationscited by 8
  21. 2021Accelerating Training of Batch Normalization: A Manifold Perspectivecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 29; 2024: 19; 2025: 25; 2026: 35.

126 papers.

Sort:

By area:

  1. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  2. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  3. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  4. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  5. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  6. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  7. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  8. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  9. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  10. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  11. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  12. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
38 more
  1. 2022Learning threshold neurons via the "edge of stability"cited by 50
  2. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  3. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  4. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  5. 2024Does SGD really happen in tiny subspaces?cited by 35
  6. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  7. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  8. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
  9. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  10. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
  11. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  12. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  13. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  14. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  15. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  16. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 17
  17. 2023Bifurcations and loss jumps in RNN trainingcited by 17
  18. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  19. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  20. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  21. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  22. 2023Loss Spike in Training Neural Networkscited by 15
  23. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  24. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  25. 2023How to escape sharp minima with random perturbationscited by 13
  26. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  27. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  28. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  29. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  30. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  31. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  32. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  33. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  34. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  35. 2024Provable Benefits of Complex Parameterizations for Structured State Space Modelscited by 9
  36. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  37. 2026The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspectivecited by 9
  38. 2022Deep Double Descent via Smooth Interpolationcited by 8

50 most cited of 126 All 126 (JSON)