mlsanityData to 14 Oct 2026

Understanding Edge-of-Stability Training Dynamics with a Minimalist Example

arXiv 2210.03294 ↗First posted 7 Oct 2022cs.LG math.OC stat.MLcited by 56

Not on the map. Why?

Abstract (arXiv)

Recently, researchers observed that gradient descent for deep neural networks operates in an ``edge-of-stability'' (EoS) regime: the sharpness (maximum eigenvalue of the Hessian) is often larger than stability threshold 2/η (where η is the step size). Despite this, the loss oscillates and converges in the long run, and the sharpness at the end is just slightly below 2/η. While many other well-understood nonconvex objectives such as matrix factorization or two-layer networks can also converge despite large sharpness, there is often a larger gap between sharpness of the endpoint and 2/η. In this paper, we study EoS phenomenon by constructing a simple function that has the same behavior. We give rigorous analysis for its training dynamics in a large local region and explain why the final converging point has sharpness close to 2/η. Globally we observe that the training dynamics for our example has an interesting bifurcating behavior, which was also observed in the training of neural nets.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 13 of its 25 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

13 of 25 references.

References · 13

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  2. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  3. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  4. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  5. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  6. 2022Understanding the unstable convergence of gradient descentcited by 86
  7. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  8. 2018A Walk with SGDcited by 78
  9. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  10. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  11. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 50
  12. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
1 more
  1. 2021Tilting the playing field: Dynamical loss functions for machine learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 15; 2024: 10; 2025: 10; 2026: 19.

56 papers.

Sort:

  1. 2022Learning threshold neurons via the "edge of stability"cited by 51
  2. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  3. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  4. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  5. 2024Does SGD really happen in tiny subspaces?cited by 36
  6. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  7. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  8. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 25
  9. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  10. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  11. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  12. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
38 more
  1. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 17
  2. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 17
  3. 2023Bifurcations and loss jumps in RNN trainingcited by 17
  4. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  5. 2023Loss Spike in Training Neural Networkscited by 16
  6. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  7. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  8. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 14
  9. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  10. 2026The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspectivecited by 10
  11. 2025Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regressioncited by 9
  12. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  13. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 9
  14. 2023On progressive sharpening, flat minima and generalisationcited by 7
  15. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 7
  16. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 7
  17. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  18. 2024From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizescited by 6
  19. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  20. 2025A Minimalist Example of Edge-of-Stability and Progressive Sharpeningcited by 4
  21. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 4
  22. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 3
  23. 2026On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentialscited by 3
  24. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  25. 2025Sharpness of Minima in Deep Matrix Factorizationcited by 2
  26. 2026Generalization at the Edge of Stabilitycited by 2
  27. 2026State-Dependent Lyapunov Analysis of Rank-1 Matrix Factorizationcited by 2
  28. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1
  29. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  30. 2026Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low Dimensioncited by 1
  31. 2026Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stabilitycited by 1
  32. 2026SGD at the Edge of Stability: The Stochastic Sharpness Gapcited by 1
  33. 2026Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attentioncited by 1
  34. 2026Mean--Fluctuation Dynamics at the Edge of Stabilitycited by 1
  35. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  36. 2024Dynamical loss functions shape landscape topography and improve learning in artificial neural networkscited by 0
  37. 2024Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regimecited by 0
  38. 2026Zeroth-Order Optimization at the Edge of Stabilitycited by 0

50 most cited of 56 All 56 (JSON)