The Origin of Edge of Stability
Not on the map. Why?
Abstract (arXiv)
Full-batch gradient descent on neural networks drives the largest Hessian eigenvalue to the threshold 2/η, where η is the learning rate. This phenomenon, the Edge of Stability, has resisted a unified explanation: existing accounts establish self-regulation near the edge but do not explain why the trajectory is forced toward 2/η from arbitrary initialization. We introduce the edge coupling, a functional on consecutive iterate pairs whose coefficient is uniquely fixed by the gradient-descent update. Differencing its criticality condition yields a step recurrence with stability boundary 2/η, and a second-order expansion yields a loss-change formula whose telescoping sum forces curvature toward 2/η. The two formulas involve different Hessian averages, but the mean value theorem localizes each to the true Hessian at an interior point of the step segment, yielding exact forcing of the Hessian eigenvalue with no gap. Setting both gradients of the edge coupling to zero classifies fixed points and period-two orbits; near a fixed point, the problem reduces to a function of the half-amplitude alone, which determines which directions support period-two orbits and on which side of the critical learning rate they appear.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 22 Apr 2026.
Builds on
17 of 27 references.
References · 17
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
- 2017Three Factors Influencing Minima in SGDcited by 270
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
- 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
- 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
- 2020Implicit Gradient Regularizationcited by 145
- 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
- 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
5 more
- 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
- 2022Understanding the unstable convergence of gradient descentcited by 84
- 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
- 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
- 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspectivecited by 1
- 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
- 2026Variational Structure at the Edge of Stabilitycited by 0
- 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0
- 2026A Journey to the Edge of Stabilitycited by 0
- 2026Muon Sublates the Edge of Stability in LLM Pretrainingcited by 0
- 2026Muon Sublates the Edge of Stability in LLM Pretraining28 Sep 2026
- 2026A Journey to the Edge of Stability26 Sep 2026
- 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow1 Sep 2026
- 2026Variational Structure at the Edge of Stability21 Aug 2026
- 2026The Road Taken: The Role of Optimizers at the Edge of Stability19 Aug 2026
- 2026Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective19 Jun 2026