mlsanityData to 14 Oct 2026

Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysis

arXiv 2506.14806 ↗First posted 3 Jun 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper establishes a continuous time approximation, a piece-wise continuous differential equation, for the discrete Heavy-Ball (HB) momentum method with explicit discretization error. Investigating continuous differential equations has been a promising approach for studying the discrete optimization methods. Despite the crucial role of momentum in gradient-based optimization methods, the gap between the original discrete dynamics and the continuous time approximations due to the discretization error has not been comprehensively bridged yet. In this work, we study the HB momentum method in continuous time while putting more focus on the discretization error to provide additional theoretical tools to this area. In particular, we design a first-order piece-wise continuous differential equation, where we add a number of counter terms to account for the discretization error explicitly. As a result, we provide a continuous time model for the HB momentum method that allows the control of discretization error to arbitrary order of the step size. As an application, we leverage it to find a new implicit regularization of the directional smoothness and investigate the implicit bias of HB for diagonal linear networks, indicating how our results can be used in deep learning. Our theoretical findings are further supported by numerical experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

29 of 43 references.

References · 29

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  3. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  4. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  5. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  6. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  7. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  8. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  9. 2020Implicit Gradient Regularizationcited by 146
  10. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  11. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  12. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
17 more
  1. 2022Understanding the unstable convergence of gradient descentcited by 86
  2. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  3. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  4. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 69
  5. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  6. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
  7. 2024The Implicit Bias of Adam on Separable Datacited by 36
  8. 2019Continuous Time Analysis of Momentum Methodscited by 32
  9. 2020The Two Regimes of Deep Network Trainingcited by 30
  10. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  11. 2016A Lyapunov Analysis of Momentum Methods in Optimizationcited by 28
  12. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  13. 2020Analysis of Stochastic Gradient Descent in Continuous Timecited by 24
  14. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 22
  15. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 13
  16. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  17. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9

Built on by

Not cited yet.