# Beyond the Edge of Stability via Two-step Gradient Updates

arXiv: https://arxiv.org/abs/2206.04172 · HTML: https://mlsanity.com/paper/2206.04172/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2022.

## Built on by (most cited first, 30 of 38)

- [Learning threshold neurons via the "edge of stability"](https://mlsanity.com/paper/2212.07469/) (2022)
- [Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency](https://mlsanity.com/paper/2402.15926/) (2024)
- [Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult](https://mlsanity.com/paper/2310.17087/) (2023)
- [Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos](https://mlsanity.com/paper/2311.02076/) (2023)
- [From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression](https://mlsanity.com/paper/2310.01687/) (2023)
- [Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization](https://mlsanity.com/paper/2406.08654/) (2024)
- [Learning Associative Memories with Gradient Descent](https://mlsanity.com/paper/2402.18724/) (2024)
- [Momentum Further Constrains Sharpness at the Edge of Stochastic Stability](https://mlsanity.com/paper/2604.14108/) (2026)
- [Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2602.01480/) (2026)
- [Benign Oscillation of Stochastic Gradient Descent with Large Learning Rates](https://mlsanity.com/paper/2310.17074/) (2023)
- [Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizes](https://mlsanity.com/paper/2504.04105/) (2025)
- [Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region](https://mlsanity.com/paper/2509.25351/) (2025)
- [From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizes](https://mlsanity.com/paper/2412.08424/) (2024)
- [A Minimalist Example of Edge-of-Stability and Progressive Sharpening](https://mlsanity.com/paper/2503.02809/) (2025)
- [Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squares](https://mlsanity.com/paper/2510.17506/) (2025)
- [On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials](https://mlsanity.com/paper/2602.14789/) (2026)
- [Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence](https://mlsanity.com/paper/2605.31484/) (2026)
- [Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2606.18080/) (2026)
- [Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks](https://mlsanity.com/paper/2501.09137/) (2025)
- [Variational Learning Finds Flatter Solutions at the Edge of Stability](https://mlsanity.com/paper/2506.12903/) (2025)
- [Generalization at the Edge of Stability](https://mlsanity.com/paper/2604.19740/) (2026)
- [Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Losses](https://mlsanity.com/paper/2502.04889/) (2025)
- [Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low Dimension](https://mlsanity.com/paper/2602.12471/) (2026)
- [Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability](https://mlsanity.com/paper/2604.02653/) (2026)
- [SGD at the Edge of Stability: The Stochastic Sharpness Gap](https://mlsanity.com/paper/2604.21016/) (2026)
- [Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima](https://mlsanity.com/paper/2607.08380/) (2026)
- [The Road Taken: The Role of Optimizers at the Edge of Stability](https://mlsanity.com/paper/2608.18415/) (2026)
- [Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regime](https://mlsanity.com/paper/2412.08025/) (2024)
- [Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Series](https://mlsanity.com/paper/2603.02620/) (2026)
- [Centre manifold theorem for maps along manifolds of fixed points](https://mlsanity.com/paper/2604.18202/) (2026)

## Abstract (arXiv)

Gradient Descent (GD) is a powerful workhorse of modern machine learning thanks to its scalability and efficiency in high-dimensional spaces. Its ability to find local minimisers is only guaranteed for losses with Lipschitz gradients, where it can be seen as a `bona-fide' discretisation of an underlying gradient flow. Yet, many ML setups involving overparametrised models do not fall into this problem class, which has motivated research beyond the so-called ``Edge of Stability'' (EoS), where the step-size crosses the admissibility threshold inversely proportional to the Lipschitz constant above. Perhaps surprisingly, GD has been empirically observed to still converge regardless of local instability and oscillatory behavior. The incipient theoretical analysis of this phenomena has mainly focused in the overparametrised regime, where the effect of choosing a large learning rate may be associated to a `Sharpness-Minimisation' implicit regularisation within the manifold of minimisers, under appropriate asymptotic limits. In contrast, in this work we directly examine the conditions for such unstable convergence, focusing on simple, yet representative, learning problems, via analysis of two-step gradient updates. Specifically, we characterize a local condition involving third-order derivatives that guarantees existence and convergence to fixed points of the two-step updates, and leverage such property in a teacher-student setting, under population loss. Finally, starting from Matrix Factorization, we provide observations of period-2 orbit of GD in high-dimensional settings with intuition of its dynamics, along with exploration into more general settings.