mlsanityData to 14 Oct 2026

Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapults

arXiv 2311.15051 ↗First posted 25 Nov 2023cs.LG math.OC stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Although gradient descent with Polyak's momentum is widely used in modern machine and deep learning, a concrete understanding of its effects on the training trajectory remains elusive. In this work, we empirically show that for linear diagonal networks and nonlinear neural networks, momentum gradient descent with a large learning rate displays large catapults, driving the iterates towards much flatter minima than those found by gradient descent. We hypothesize that the large catapult is caused by momentum "prolonging" the self-stabilization effect (Damian et al., 2023). We provide theoretical and empirical support for our hypothesis in a simple toy example and empirical evidence supporting our hypothesis for linear diagonal networks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 25 Nov 2023.

Builds on

37 of 53 references.

References · 37

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  3. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  4. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  6. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  7. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  8. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  9. 2020Implicit Gradient Regularizationcited by 146
  10. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  11. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  12. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
25 more
  1. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  2. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  3. 2022Understanding the unstable convergence of gradient descentcited by 86
  4. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 71
  5. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  6. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  7. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  8. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  9. 2022Learning threshold neurons via the "edge of stability"cited by 51
  10. 2018On the insufficiency of existing momentum schemes for Stochastic Optimizationcited by 49
  11. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  12. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  13. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  14. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  15. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  16. 2020Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectivescited by 27
  17. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  18. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  19. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 25
  20. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 22
  21. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  22. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  23. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
  24. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  25. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  2. 2026Towards Understanding Momentum Acceleration in River-Valley Loss Landscapecited by 1
  3. 2026Variational Structure at the Edge of Stabilitycited by 0