mlsanityData to 7 Oct 2026

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

arXiv 2403.05293 ↗First posted 8 Mar 2024cs.LG math.OC stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size γ and momentum parameter β that allows us to identify an intrinsic quantity λ = frac γ (1 - β)2 which uniquely defines the optimisation path and provides a simple acceleration rule. When training a 2-layer diagonal linear network in an overparametrised regression setting, we characterise the recovered solution through an implicit regularisation problem. We then prove that small values of λ help to recover sparse solutions. Finally, we give similar but weaker results for stochastic momentum gradient descent. We provide numerical experiments which support our claims.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 34 of its 53 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

34 of 53 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  7. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  8. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  9. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  10. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  11. 2020Momentum Improves Normalized SGDcited by 136
  12. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 117
22 more
  1. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 108
  2. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  3. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  4. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  5. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 68
  6. 2017Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descentcited by 66
  7. 2014Global convergence of the Heavy-ball method for convex optimizationcited by 60
  8. 2018On the insufficiency of existing momentum schemes for Stochastic Optimizationcited by 46
  9. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  10. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  11. 2019Exponentiated Gradient Meets Gradient Descentcited by 32
  12. 2019Continuous Time Analysis of Momentum Methodscited by 32
  13. 2020Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimizationcited by 32
  14. 2020The Two Regimes of Deep Network Trainingcited by 30
  15. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  16. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  17. 2015From Averaging to Acceleration, There is Only a Step-sizecited by 24
  18. 2019Accelerated Linear Convergence of Stochastic Momentum Methods in Wasserstein Distancescited by 15
  19. 2019Heavy-ball Algorithms Always Escape Saddle Pointscited by 13
  20. 2020The connections between Lyapunov functions for some optimization algorithms and differential equationscited by 11
  21. 2021Convergence rates for the Heavy-Ball continuous dynamics for non-convex optimization, under Polyak-L ojasiewicz conditioncited by 11
  22. 2019The Role of Memory in Stochastic Optimizationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 3.

9 papers.

  1. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47
  2. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  3. 2024Implicit Bias of Mirror Flow on Separable Datacited by 7
  4. 2024Optimization Insights into Deep Diagonal Linear Networkscited by 2
  5. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 2
  6. 2025Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theorycited by 1
  7. 2025Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysiscited by 0
  8. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0
  9. 2026Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initializationcited by 0