# Edge of Stochastic Stability: Revisiting the Edge of Stability for SGD

arXiv: https://arxiv.org/abs/2412.20553 · HTML: https://mlsanity.com/paper/2412.20553/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2024.

## Built on by (most cited first, 30 of 30)

- [A Defense of the Quadratic Model](https://mlsanity.com/paper/2607.21716/) (2026)
- [Momentum Further Constrains Sharpness at the Edge of Stochastic Stability](https://mlsanity.com/paper/2604.14108/) (2026)
- [Non-Euclidean Gradient Descent Operates at the Edge of Stability](https://mlsanity.com/paper/2603.05002/) (2026)
- [Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer](https://mlsanity.com/paper/2605.07870/) (2026)
- [A Dynamical Systems Perspective on the Analysis of Neural Networks](https://mlsanity.com/paper/2507.05164/) (2025)
- [Too Sharp, Too Sure: When Calibration Follows Curvature](https://mlsanity.com/paper/2604.20614/) (2026)
- [Variational Learning Finds Flatter Solutions at the Edge of Stability](https://mlsanity.com/paper/2506.12903/) (2025)
- [Generalization at the Edge of Stability](https://mlsanity.com/paper/2604.19740/) (2026)
- [Optimizer Dynamics at the Edge of Stability with Differential Privacy](https://mlsanity.com/paper/2512.19019/) (2025)
- [Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model](https://mlsanity.com/paper/2602.05065/) (2026)
- [Large Spikes in Stochastic Gradient Descent: A Large-Deviations View](https://mlsanity.com/paper/2603.10079/) (2026)
- [Does Weight Decay Enhance Training Stability?](https://mlsanity.com/paper/2605.16622/) (2026)
- [Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization](https://mlsanity.com/paper/2606.30226/) (2026)
- [The Road Taken: The Role of Optimizers at the Edge of Stability](https://mlsanity.com/paper/2608.18415/) (2026)
- [Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal](https://mlsanity.com/paper/2608.20638/) (2026)
- [Feature Learning Beyond the Edge of Stability](https://mlsanity.com/paper/2502.13110/) (2025)
- [Convergence of optimizers implies eigenvalues filtering at equilibrium](https://mlsanity.com/paper/2510.09034/) (2025)
- [Stability as a Liability:Systematic Breakdown of Linguistic Structure in LLMs](https://mlsanity.com/paper/2601.18588/) (2026)
- [Why Some Models Resist Unlearning: A Linear Stability Perspective](https://mlsanity.com/paper/2602.02986/) (2026)
- [Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Series](https://mlsanity.com/paper/2603.02620/) (2026)
- [Zeroth-Order Optimization at the Edge of Stability](https://mlsanity.com/paper/2604.14669/) (2026)
- [Edge of Stability Selectively Shapes Learning Across the Data Distribution](https://mlsanity.com/paper/2606.04212/) (2026)
- [Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam](https://mlsanity.com/paper/2607.03998/) (2026)
- [The Fourth Quadrant: A Stylized View of Benign Misfitting](https://mlsanity.com/paper/2608.01032/) (2026)
- [Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining](https://mlsanity.com/paper/2608.28442/) (2026)
- [A Journey to the Edge of Stability](https://mlsanity.com/paper/2609.32290/) (2026)
- [Muon Sublates the Edge of Stability in LLM Pretraining](https://mlsanity.com/paper/2609.34915/) (2026)
- [Tight Transition Time Bounds for Separable Logistic Regression at the Edge of Stability](https://mlsanity.com/paper/2610.01459/) (2026)
- [Broken scale symmetries in undercomplete linear autoencoders](https://mlsanity.com/paper/2610.03640/) (2026)
- [A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling](https://mlsanity.com/paper/2610.06701/) (2026)

## Abstract (arXiv)

Recent findings by Cohen et al., 2021, demonstrate that when training neural networks using full-batch gradient descent with a step size of η, the largest eigenvalue λmax of the full-batch Hessian consistently stabilizes around 2/η. These results have significant implications for convergence and generalization. This, however, is not the case for mini-batch optimization algorithms, limiting the broader applicabilityof the consequences of these findings. We show mini-batch Stochastic Gradient Descent (SGD) trains in a different regime we term Edge of Stochastic Stability (EoSS). In this regime, what stabilizes at 2/η is Batch Sharpness: the expected directional curvature of mini-batch Hessians along their corresponding stochastic gradients. As a consequence λmax -- which is generally smaller than Batch Sharpness -- is suppressed, aligning with the long-standing empirical observation that smaller batches and larger step sizes favor flatter minima. We further discuss implications for mathematical modeling of SGD trajectories.