mlsanityData to 14 Oct 2026

The Road Taken: The Role of Optimizers at the Edge of Stability

arXiv 2608.18415 ↗First posted 19 Aug 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

The edge of stability refers to a phenomenon in deep learning with gradient-based optimizers where the Hessian eigenvalues of the loss remain stable above a threshold that the classical descent lemma predicts to be unstable. Previous works formulate the edge of stability with respect to the maximum Hessian eigenvalue and the learning rate. However, we observe that many first-order methods, including gradient descent, significantly violate the stability bound predicted by these theories by a factor as large as × 21.1. Moreover, this deviation turns out to be systematic and highly dependent on the underlying optimizer, which is not captured by previous formulations. This calls for a new formulation of the stability threshold, which we derive from the directional Hessian and the gradient-alignment score with respect to the actual update taken by the optimizer, rather than the maximum curvature mode. Our new formulation of the realized edge of stability not only removes optimizer-dependent offsets and provides more consistent predictions of the stability threshold, but also introduces new diagnostic tools that reveal the unique role of the optimizer in actively balancing between the temporal and spatial budgets in first-order optimization.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 46 of its 62 references that are in the map. First posted on arXiv 19 Aug 2026.

Builds on

46 of 62 references.

References · 46

  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  5. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  6. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  7. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  8. 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
  9. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  10. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  11. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
  12. 2022Understanding the unstable convergence of gradient descentcited by 86
34 more
  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2018A Walk with SGDcited by 78
  3. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  4. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  5. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
  6. 2022Learning threshold neurons via the "edge of stability"cited by 51
  7. 2018Quasi-hyperbolic momentum and Adam for deep learningcited by 48
  8. 2019Understanding the Role of Momentum in Stochastic Gradient Methodscited by 44
  9. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
  10. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  11. 2024Grokfast: Accelerated Grokking by Amplifying Slow Gradientscited by 41
  12. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  13. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  14. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  15. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  16. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  17. 2024Edge of Stochastic Stability: Revisiting the Edge of Stability for SGDcited by 30
  18. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  19. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 25
  20. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  21. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 17
  22. 2024Directional Smoothness and Gradient Methods: Convergence and Adaptivitycited by 17
  23. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 10
  24. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 9
  25. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 9
  26. 2017Control Interpretations for First-Order Optimization Methodscited by 8
  27. 2024On the Performance Analysis of Momentum Method: A Frequency Domain Perspectivecited by 8
  28. 2025Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stabilitycited by 6
  29. 2026The Origin of Edge of Stabilitycited by 6
  30. 2026On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentialscited by 3
  31. 2026A Rod Flow Model for Adam at the Edge of Stabilitycited by 3
  32. 2026Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 3
  33. 2025Greedy Alignment Principle for Optimizer Selectioncited by 2
  34. 2026The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical Systemcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Journey to the Edge of Stabilitycited by 0