On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials
Not on the map. Why?
Abstract (arXiv)
The dynamical stability of the iterates during training plays a key role in determining the minima obtained by optimization algorithms. For example, stable solutions of gradient descent (GD) correspond to flat minima, which have been associated with favorable features. While prior work often relies on linearization to determine stability, it remains unclear whether linearized dynamics faithfully capture the full nonlinear behavior. Recent work has shown that GD may stably oscillate near a linearly unstable minimum and still converge once the step size decays, indicating that linear analysis can be misleading. In this work, we explicitly study the effect of nonlinear terms. Specifically, we derive an exact criterion for stable oscillations of GD near minima in the multivariate setting. Our condition depends on high-order derivatives, generalizing existing results. Extending the analysis to stochastic gradient descent (SGD), we show that nonlinear dynamics can diverge in expectation even if a single batch is unstable. This implies that stability can be dictated by a single batch that oscillates unstably, rather than an average effect, as linear analysis suggests. Finally, we prove that if all batches are linearly stable, the nonlinear dynamics of SGD are stable in expectation.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 16 Feb 2026.
Builds on
23 of 37 references.
References · 23
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
- 2017Three Factors Influencing Minima in SGDcited by 273
- 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
- 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
- 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
- 2022Understanding the unstable convergence of gradient descentcited by 86
- 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
- 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
- 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
- 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
11 more
- 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
- 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
- 2021Strength of Minibatch Noise in SGDcited by 39
- 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
- 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 37
- 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
- 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
- 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 17
- 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
- 2022A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momentacited by 11
- 2023Exact Mean Square Linear Stability Analysis for SGDcited by 11
Built on by
Citing papers per year: 2026: 3.
3 papers.