# Understanding Edge-of-Stability Training Dynamics with a Minimalist Example

arXiv: https://arxiv.org/abs/2210.03294 · HTML: https://mlsanity.com/paper/2210.03294/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [The large learning rate phase of deep learning: the catapult mechanism](https://mlsanity.com/paper/2003.02218/) (2020)
- [The Break-Even Point on Optimization Trajectories of Deep Neural Networks](https://mlsanity.com/paper/2002.09572/) (2020)
- [Understanding Gradient Descent on Edge of Stability in Deep Learning](https://mlsanity.com/paper/2205.09745/) (2022)
- [Understanding the unstable convergence of gradient descent](https://mlsanity.com/paper/2204.01050/) (2022)
- [Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction](https://mlsanity.com/paper/2206.07085/) (2022)
- [A Walk with SGD](https://mlsanity.com/paper/1802.08770/) (2018)
- [Theoretical Analysis of Auto Rate-Tuning by Batch Normalization](https://mlsanity.com/paper/1812.03981/) (2018)
- [Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability](https://mlsanity.com/paper/2207.12678/) (2022)
- [Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect](https://mlsanity.com/paper/2110.03677/) (2021)
- [Robust Training of Neural Networks Using Scale Invariant Architectures](https://mlsanity.com/paper/2202.00980/) (2022)
- [Tilting the playing field: Dynamical loss functions for machine learning](https://mlsanity.com/paper/2102.03793/) (2021)

## Built on by (most cited first, 30 of 56)

- [Learning threshold neurons via the "edge of stability"](https://mlsanity.com/paper/2212.07469/) (2022)
- [Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability](https://mlsanity.com/paper/2305.11788/) (2023)
- [Second-order regression models exhibit progressive sharpening to the edge of stability](https://mlsanity.com/paper/2210.04860/) (2022)
- [(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability](https://mlsanity.com/paper/2302.08982/) (2023)
- [Does SGD really happen in tiny subspaces?](https://mlsanity.com/paper/2405.16002/) (2024)
- [Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency](https://mlsanity.com/paper/2402.15926/) (2024)
- [Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory](https://mlsanity.com/paper/2307.04204/) (2023)
- [Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult](https://mlsanity.com/paper/2310.17087/) (2023)
- [Super Consistency of Neural Network Landscapes and Learning Rate Transfer](https://mlsanity.com/paper/2402.17457/) (2024)
- [Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos](https://mlsanity.com/paper/2311.02076/) (2023)
- [Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond](https://mlsanity.com/paper/2305.13064/) (2023)
- [When and Why Momentum Accelerates SGD:An Empirical Study](https://mlsanity.com/paper/2306.09000/) (2023)
- [Sharpness-Aware Minimization and the Edge of Stability](https://mlsanity.com/paper/2309.12488/) (2023)
- [From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression](https://mlsanity.com/paper/2310.01687/) (2023)
- [Bifurcations and loss jumps in RNN training](https://mlsanity.com/paper/2310.17561/) (2023)
- [Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization](https://mlsanity.com/paper/2406.08654/) (2024)
- [Loss Spike in Training Neural Networks](https://mlsanity.com/paper/2305.12133/) (2023)
- [Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks](https://mlsanity.com/paper/2305.12467/) (2023)
- [A Universal Class of Sharpness-Aware Minimization Algorithms](https://mlsanity.com/paper/2406.03682/) (2024)
- [Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes](https://mlsanity.com/paper/2406.05033/) (2024)
- [Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization](https://mlsanity.com/paper/2311.04163/) (2023)
- [The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective](https://mlsanity.com/paper/2602.00170/) (2026)
- [Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regression](https://mlsanity.com/paper/2506.02336/) (2025)
- [Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training](https://mlsanity.com/paper/2507.09846/) (2025)
- [Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2602.01480/) (2026)
- [On progressive sharpening, flat minima and generalisation](https://mlsanity.com/paper/2305.14683/) (2023)
- [Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizes](https://mlsanity.com/paper/2504.04105/) (2025)
- [Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region](https://mlsanity.com/paper/2509.25351/) (2025)
- [Understanding the Generalization Benefits of Late Learning Rate Decay](https://mlsanity.com/paper/2401.11600/) (2024)
- [From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizes](https://mlsanity.com/paper/2412.08424/) (2024)

## Abstract (arXiv)

Recently, researchers observed that gradient descent for deep neural networks operates in an ``edge-of-stability'' (EoS) regime: the sharpness (maximum eigenvalue of the Hessian) is often larger than stability threshold 2/η (where η is the step size). Despite this, the loss oscillates and converges in the long run, and the sharpness at the end is just slightly below 2/η. While many other well-understood nonconvex objectives such as matrix factorization or two-layer networks can also converge despite large sharpness, there is often a larger gap between sharpness of the endpoint and 2/η. In this paper, we study EoS phenomenon by constructing a simple function that has the same behavior. We give rigorous analysis for its training dynamics in a large local region and explain why the final converging point has sharpness close to 2/η. Globally we observe that the training dynamics for our example has an interesting bifurcating behavior, which was also observed in the training of neural nets.