mlsanityData to 7 Oct 2026

The Origin of Edge of Stability

arXiv 2604.20446 ↗First posted 22 Apr 2026cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Full-batch gradient descent on neural networks drives the largest Hessian eigenvalue to the threshold 2/η, where η is the learning rate. This phenomenon, the Edge of Stability, has resisted a unified explanation: existing accounts establish self-regulation near the edge but do not explain why the trajectory is forced toward 2/η from arbitrary initialization. We introduce the edge coupling, a functional on consecutive iterate pairs whose coefficient is uniquely fixed by the gradient-descent update. Differencing its criticality condition yields a step recurrence with stability boundary 2/η, and a second-order expansion yields a loss-change formula whose telescoping sum forces curvature toward 2/η. The two formulas involve different Hessian averages, but the mean value theorem localizes each to the true Hessian at an interior point of the step segment, yielding exact forcing of the Hessian eigenvalue with no gap. Setting both gradients of the edge coupling to zero classifies fixed points and period-two orbits; near a fixed point, the problem reduces to a function of the half-amplitude alone, which determines which directions support period-two orbits and on which side of the critical learning rate they appear.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

17 of 27 references.

References · 17

  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  4. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  5. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  6. 2017Three Factors Influencing Minima in SGDcited by 270
  7. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  8. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  9. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  10. 2020Implicit Gradient Regularizationcited by 145
  11. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  12. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
5 more
  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2022Understanding the unstable convergence of gradient descentcited by 84
  3. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  4. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  5. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.