mlsanityData to 7 Oct 2026

Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability

arXiv 2103.00065 ↗First posted 26 Feb 2021cs.LG stat.MLcited by 413

Not on the map. Why?

Abstract (arXiv)

We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value 2 / (step size), and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability. Code is available at https://github.com/locuslab/edge-of-stability.

Key facts. Cited by 413 papers in the map (data through 7 Oct 2026). Builds on 41 of its 67 references that are in the map. First posted on arXiv 26 Feb 2021.

Builds on

41 of 67 references.

References · 41

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  11. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  12. 2018How Does Batch Normalization Help Optimization?cited by 355
29 more
  1. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  2. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  3. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  4. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  5. 2017Three Factors Influencing Minima in SGDcited by 270
  6. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  7. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  8. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  9. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  10. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  11. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  12. 2020Implicit Gradient Regularizationcited by 145
  13. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 143
  14. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  15. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  16. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  17. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  18. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  19. 2018A Walk with SGDcited by 74
  20. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  21. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  22. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  23. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  24. 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
  25. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  26. 2018Step Size Matters in Deep Learningcited by 19
  27. 2019The asymptotic spectrum of the Hessian of DNN throughout trainingcited by 19
  28. 2019At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?cited by 16
  29. 2019Neural Spectrum Alignment: Empirical Studycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 39; 2023: 72; 2024: 65; 2025: 93; 2026: 129.

413 papers.

Sort:

By area:

  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  2. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  3. 2023Understanding plasticity in neural networkscited by 170
  4. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  5. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  6. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  7. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  8. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  9. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  10. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  11. 2024Scaling Laws for Precisioncited by 92
  12. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
38 more
  1. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  2. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  3. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  4. 2022Understanding the unstable convergence of gradient descentcited by 84
  5. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  6. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  7. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  8. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  9. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  10. 2024Scalable Optimization in the Modular Normcited by 64
  11. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  12. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  13. 2021Handcrafted Backdoors in Deep Neural Networkscited by 57
  14. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  15. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  16. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  17. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  18. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  19. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  20. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  21. 2022Learning threshold neurons via the "edge of stability"cited by 50
  22. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  23. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  24. 2024Why Warmup the Learning Rate? Underlying Mechanisms and Improvementscited by 45
  25. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  26. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  27. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  28. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  29. 2025Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networkscited by 41
  30. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  31. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  32. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
  33. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  34. 2025Model Merging in Pre-training of Large Language Modelscited by 36
  35. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  36. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  37. 2024Does SGD really happen in tiny subspaces?cited by 35
  38. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34

50 most cited of 413 All 413 (JSON)