Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability
Not on the map. Why?
Abstract (arXiv)
How can we understand gradient-based training over non-convex landscapes? The edge of stability phenomenon, introduced in Cohen et al. (2021), indicates that the answer is not so simple: namely, gradient descent (GD) with large step sizes often diverges away from the gradient flow. In this regime, the "Central Flow", recently proposed in Cohen et al. (2025), provides an accurate ODE approximation to the GD dynamics over many architectures. In this work, we propose Rod Flow, an alternative ODE approximation, which carries the following advantages: (1) it rests on a principled derivation stemming from a physical picture of GD iterates as an extended one-dimensional object -- a "rod"; (2) it better captures GD dynamics for simple toy examples and matches the accuracy of Central Flow for representative neural network architectures, and (3) is explicit and cheap to compute. Theoretically, we prove that Rod Flow correctly predicts the critical sharpness threshold and explains self-stabilization in quartic potentials. We validate our theory with a range of numerical experiments.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 21 of its 24 references that are in the map. First posted on arXiv 1 Feb 2026.
Builds on
21 of 24 references.
References · 21
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
- 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
- 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
- 2020Implicit Gradient Regularizationcited by 145
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
- 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
- 2022Understanding the unstable convergence of gradient descentcited by 84
- 2024Understanding Optimization in Deep Learning with Central Flowscited by 57
- 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
- 2022Learning threshold neurons via the "edge of stability"cited by 50
- 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
9 more
- 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
- 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
- 2019Continuous Time Analysis of Momentum Methodscited by 32
- 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
- 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
- 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
- 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
- 2025Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stabilitycited by 6
- 2023Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descentcited by 5
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 8
- 2026Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 3
- 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
- 2026Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attentioncited by 0
- 2026Variational Structure at the Edge of Stabilitycited by 0
- 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0
- 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow1 Sep 2026
- 2026Variational Structure at the Edge of Stability21 Aug 2026
- 2026The Road Taken: The Role of Optimizers at the Edge of Stability19 Aug 2026
- 2026Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stability16 Jun 2026
- 2026Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attention20 May 2026
- 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging3 Feb 2026