# Understanding Optimization in Deep Learning with Central Flows

arXiv: https://arxiv.org/abs/2410.24206 · HTML: https://mlsanity.com/paper/2410.24206/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2024.

## Built on by (most cited first, 30 of 64)

- [Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?](https://mlsanity.com/paper/2511.00674/) (2025)
- [The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training](https://mlsanity.com/paper/2502.19002/) (2025)
- [A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules](https://mlsanity.com/paper/2503.12811/) (2025)
- [Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks](https://mlsanity.com/paper/2507.02119/) (2025)
- [What Really Matters in Matrix-Whitening Optimizers?](https://mlsanity.com/paper/2510.25000/) (2025)
- [A Defense of the Quadratic Model](https://mlsanity.com/paper/2607.21716/) (2026)
- [Adaptive Preconditioners Trigger Loss Spikes in Adam](https://mlsanity.com/paper/2506.04805/) (2025)
- [Momentum Further Constrains Sharpness at the Edge of Stochastic Stability](https://mlsanity.com/paper/2604.14108/) (2026)
- [Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning](https://mlsanity.com/paper/2505.12387/) (2025)
- [Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regression](https://mlsanity.com/paper/2506.02336/) (2025)
- [Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training](https://mlsanity.com/paper/2507.09846/) (2025)
- [Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2602.01480/) (2026)
- [Non-Euclidean Gradient Descent Operates at the Edge of Stability](https://mlsanity.com/paper/2603.05002/) (2026)
- [GradPower: Powering Gradients for Faster Language Model Pre-Training](https://mlsanity.com/paper/2505.24275/) (2025)
- [A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs](https://mlsanity.com/paper/2601.16979/) (2026)
- [Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory](https://mlsanity.com/paper/2602.09314/) (2026)
- [Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement](https://mlsanity.com/paper/2602.22681/) (2026)
- [Understanding the Mechanisms of Fast Hyperparameter Transfer](https://mlsanity.com/paper/2512.22768/) (2025)
- [Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay](https://mlsanity.com/paper/2605.26097/) (2026)
- [Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability](https://mlsanity.com/paper/2507.12837/) (2025)
- [(How) Learning Rates Regulate Catastrophic Overtraining](https://mlsanity.com/paper/2604.13627/) (2026)
- [Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More](https://mlsanity.com/paper/2506.06940/) (2025)
- [Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squares](https://mlsanity.com/paper/2510.17506/) (2025)
- [Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima](https://mlsanity.com/paper/2604.09258/) (2026)
- [Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes](https://mlsanity.com/paper/2605.06152/) (2026)
- [AMUSE: Anytime Muon with Stable Gradient Evaluation](https://mlsanity.com/paper/2605.22432/) (2026)
- [Hyperball May Not Be a Free Lunch](https://mlsanity.com/paper/2607.22444/) (2026)
- [Generalization Below the Edge of Stability: The Role of Data Geometry](https://mlsanity.com/paper/2510.18120/) (2025)
- [Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2606.18080/) (2026)
- [Variational Learning Finds Flatter Solutions at the Edge of Stability](https://mlsanity.com/paper/2506.12903/) (2025)

## Abstract (arXiv)

Traditional theories of optimization cannot describe the dynamics of optimization in deep learning, even in the simple setting of deterministic training. The challenge is that optimizers typically operate in a complex, oscillatory regime called the "edge of stability." In this paper, we develop theory that can describe the dynamics of optimization in this regime. Our key insight is that while the *exact* trajectory of an oscillatory optimizer may be challenging to analyze, the *time-averaged* (i.e. smoothed) trajectory is often much more tractable. To analyze an optimizer, we derive a differential equation called a "central flow" that characterizes this time-averaged trajectory. We empirically show that these central flows can predict long-term optimization trajectories for generic neural networks with a high degree of numerical accuracy. By interpreting these central flows, we are able to understand how gradient descent makes progress even as the loss sometimes goes up; how adaptive optimizers "adapt" to the local loss landscape; and how adaptive optimizers implicitly navigate towards regions where they can take larger steps. Our results suggest that central flows can be a valuable theoretical tool for reasoning about optimization in deep learning.