mlsanityData to 7 Oct 2026

Understanding the unstable convergence of gradient descent

arXiv 2204.01050 ↗First posted 3 Apr 2022math.OC cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

Most existing analyses of (stochastic) gradient descent rely on the condition that for L-smooth costs, the step size is less than 2/L. However, many works have observed that in machine learning applications step sizes often do not fulfill this condition, yet (stochastic) gradient descent still converges, albeit in an unstable manner. We investigate this unstable convergence phenomenon from first principles, and discuss key causes behind it. We also identify its main characteristics, and how they interrelate based on both theory and experiments, offering a principled view toward understanding the phenomenon.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 9 of its 14 references that are in the map. First posted on arXiv 3 Apr 2022.

Builds on

9 of 14 references.

References · 9

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  4. 2017Three Factors Influencing Minima in SGDcited by 270
  5. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  6. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  7. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  8. 2018A Walk with SGDcited by 74
  9. 2016Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regionscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 20; 2024: 10; 2025: 16; 2026: 23.

84 papers.

Sort:

  1. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  2. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  3. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  4. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  5. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  6. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  7. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  8. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  9. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  10. 2022Learning threshold neurons via the "edge of stability"cited by 50
  11. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  12. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
38 more
  1. 2024Does SGD really happen in tiny subspaces?cited by 35
  2. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 34
  3. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  4. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  5. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
  6. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  7. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
  8. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  9. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  10. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  11. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 17
  12. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  13. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  14. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  15. 2023Loss Spike in Training Neural Networkscited by 15
  16. 2024Directional Smoothness and Gradient Methods: Convergence and Adaptivitycited by 15
  17. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  18. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  19. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  20. 2022On the generalization of learning algorithms that do not convergecited by 9
  21. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 9
  22. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 8
  23. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 8
  24. 2023On progressive sharpening, flat minima and generalisationcited by 6
  25. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 6
  26. 2026The Origin of Edge of Stabilitycited by 6
  27. 2026A Defense of the Quadratic Modelcited by 6
  28. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
  29. 2023Learning Trajectories are Generalization Indicatorscited by 5
  30. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 5
  31. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  32. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  33. 2025A Minimalist Example of Edge-of-Stability and Progressive Sharpeningcited by 4
  34. 2025On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approachcited by 4
  35. 2025A Dynamical Systems Perspective on the Analysis of Neural Networkscited by 3
  36. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 3
  37. 2023Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testingcited by 2
  38. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2

50 most cited of 84 All 84 (JSON)