mlsanityData to 14 Oct 2026

From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression

arXiv 2310.01687 ↗First posted 2 Oct 2023cs.LG math.DS math.OCcited by 17

Not on the map. Why?

Abstract (arXiv)

We conduct a comprehensive investigation into the dynamics of gradient descent using large-order constant step-sizes in the context of quadratic regression models. Within this framework, we reveal that the dynamics can be encapsulated by a specific cubic map, naturally parameterized by the step-size. Through a fine-grained bifurcation analysis concerning the step-size parameter, we delineate five distinct training phases: (1) monotonic, (2) catapult, (3) periodic, (4) chaotic, and (5) divergent, precisely demarcating the boundaries of each phase. As illustrations, we provide examples involving phase retrieval and two-layer neural networks employing quadratic activation functions and constant outer-layers, utilizing orthogonal training data. Our simulations indicate that these five phases also manifest with generic non-orthogonal data. We also empirically investigate the generalization performance when training in the various non-monotonic (and non-divergent) phases. In particular, we observe that performing an ergodic trajectory averaging stabilizes the test error in non-monotonic (and non-divergent) phases.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 39 of its 72 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

39 of 72 references.

References · 39

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  5. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  6. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  7. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  8. 2022Understanding the unstable convergence of gradient descentcited by 86
  9. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  10. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  11. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  12. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
27 more
  1. 2022Learning threshold neurons via the "edge of stability"cited by 51
  2. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 50
  3. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  4. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  5. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  6. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  7. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  8. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  9. 2020Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Functioncited by 26
  10. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  11. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  12. 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
  13. 2020The Numerics of Phase Retrievalcited by 21
  14. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  15. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  16. 2023Provably Faster Gradient Descent via Long Stepscited by 20
  17. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  18. 2023Acceleration by Stepsize Hedging I: Multi-Step Descent and the Silver Stepsize Schedulecited by 19
  19. 2021Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspectivecited by 15
  20. 2023Generalization Bounds with Data-dependent Fractal Dimensionscited by 15
  21. 2021Super-Acceleration with Cyclical Step-sizescited by 14
  22. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  23. 2021Provable Super-Convergence with a Large Cyclical Learning Ratecited by 10
  24. 2022Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descentcited by 9
  25. 2022On the generalization of learning algorithms that do not convergecited by 9
  26. 2023Accelerated Gradient Descent via Long Stepscited by 9
  27. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 4; 2026: 6.

17 papers.

Sort:

  1. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  2. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  3. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 14
  4. 2024The boundary of neural network trainability is fractalcited by 13
  5. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 9
  6. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 7
  7. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 7
  8. 2025Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descentcited by 4
  9. 2026On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentialscited by 3
  10. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  11. 2026The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical Systemcited by 2
  12. 2024Complex fractal trainability boundary can arise from trivial non-convexitycited by 1
5 more
  1. 2024Tracking solutions of time-varying variational inequalitiescited by 1
  2. 2026Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attentioncited by 1
  3. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  4. 2024Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regimecited by 0
  5. 2026A Journey to the Edge of Stabilitycited by 0