mlsanityData to 14 Oct 2026

The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow

arXiv 2609.01034 ↗First posted 1 Sep 2026cs.LG math.OC stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The central flow of Cohen et al. (2025) is an empirically accurate continuous-time model of gradient descent at the edge of stability in deep learning, However, its derivation is heuristic. We propose a perturbative regime in which the central flow is the limit of gradient descent: we assume that the loss decomposes as f = g + varepsilon h; in the limit varepsilon → 0, the dynamics of gradient descent with learning rate η converge to the gradient flow of h constrained to the minimizers of g of sharpness at most 2/η. Our approach is formal rather than rigorous; it treats gradient descent as a singularly perturbed dynamical system in varepsilon. Three timescales emerge: a fast timescale of oscillations along the sharpest direction, an intermediate timescale of the self-stabilization mechanism, and a slow timescale of the dynamics along the minimizers of g-the central flow. Using the method of multiple scales, a classical formal method from singular perturbation theory, we derive the expansion of the dynamics in varepsilon: the central flow emerges as the leading-order term in the expansion, while the self-stabilization mechanism appears in the next-order term. We study this mechanism beyond previous analyses: with a single eigenvalue at the edge of stability, we compute the slow drift of the energy of the fluctuations; with several eigenvalues at the edge of stability, we derive the self-stabilization system and explain why fluctuations persist.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 45 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

42 of 45 references.

References · 42

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  2. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  3. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  4. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  5. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  6. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  7. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  8. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  9. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  10. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
  11. 2022Understanding the unstable convergence of gradient descentcited by 86
  12. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
30 more
  1. 2018A Walk with SGDcited by 78
  2. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  3. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  4. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  5. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  6. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
  7. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  8. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  9. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  10. 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
  11. 2024Does SGD really happen in tiny subspaces?cited by 36
  12. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  13. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  14. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 25
  15. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  16. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  17. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  18. 2025Neural Thermodynamic Laws for Large Language Model Trainingcited by 10
  19. 2025Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regressioncited by 9
  20. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 9
  21. 2025FOCUS: First Order Concentrated Updating Schemecited by 8
  22. 2026The Origin of Edge of Stabilitycited by 6
  23. 2025Learning Dynamics of Deep Linear Networks Beyond the Edge of Stabilitycited by 5
  24. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  25. 2025A Minimalist Example of Edge-of-Stability and Progressive Sharpeningcited by 4
  26. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 4
  27. 2026Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 3
  28. 2026Mean--Fluctuation Dynamics at the Edge of Stabilitycited by 1
  29. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  30. 2026Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuationscited by 1

Built on by

Not cited yet.