mlsanityData to 14 Oct 2026

Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region

arXiv 2509.25351 ↗First posted 29 Sep 2025cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

We examine gradient descent in matrix factorization and show that under large step sizes the parameter space develops a fractal structure. We derive the exact critical step size for convergence in scalar-vector factorization and show that near criticality the selected minimizer depends sensitively on the initialization. Moreover, we show that adding regularization amplifies this sensitivity, generating a fractal boundary between initializations that converge and those that diverge. The analysis extends to general matrix factorization with orthogonal initialization. Our findings reveal that near-critical step sizes induce a chaotic regime of gradient descent where the training outcome is unpredictable and there are no simple implicit biases, such as towards balancedness, minimum norm, or flatness.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 55 of its 99 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

55 of 99 references.

References · 55

  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  2. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  3. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  5. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  6. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  7. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  8. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  9. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  10. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  11. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 129
  12. 2016Identity Matters in Deep Learningcited by 109
38 more
  1. 2015Low-rank Solutions of Linear Matrix Equations via Procrustes Flowcited by 97
  2. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  3. 2022Understanding the unstable convergence of gradient descentcited by 86
  4. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  5. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  6. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  7. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  8. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  9. 2022Learning threshold neurons via the "edge of stability"cited by 51
  10. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 50
  11. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  12. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 45
  13. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  14. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  15. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  16. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  17. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  18. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  19. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  20. 2020Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Functioncited by 26
  21. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  22. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 25
  23. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  24. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  25. 2018Step Size Matters in Deep Learningcited by 19
  26. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  27. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  28. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 17
  29. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  30. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  31. 2019Global Convergence of Gradient Descent for Deep Linear Residual Networkscited by 16
  32. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  33. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 14
  34. 2024The boundary of neural network trainability is fractalcited by 13
  35. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  36. 2024The geometry of the deep linear networkcited by 11
  37. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8
  38. 2024Dynamical stability and chaos in artificial neural network trajectories along trainingcited by 6

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025Sharpness of Minima in Deep Matrix Factorizationcited by 2
  2. 2026State-Dependent Lyapunov Analysis of Rank-1 Matrix Factorizationcited by 2
  3. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 1
  4. 2026Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stabilitycited by 1
  5. 2026Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attentioncited by 1
  6. 2026A Bifurcation Theory Framework for Gradient Descent on the Edge of Stabilitycited by 0
  7. 2026Mind the Drift: Diagonal Linear Networks Under Large Learning Ratescited by 0