mlsanityData to 7 Oct 2026

Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent

arXiv 2302.00849 ↗First posted 2 Feb 2023cs.LG math.OCcited by 25

Not on the map. Why?

Abstract (arXiv)

It is well known that the finite step-size (h) in Gradient Descent (GD) implicitly regularizes solutions to flatter minima. A natural question to ask is "Does the momentum parameter β play a role in implicit regularization in Heavy-ball (H.B) momentum accelerated gradient descent (GD+M)?". To answer this question, first, we show that the discrete H.B momentum update (GD+M) follows a continuous trajectory induced by a modified loss, which consists of an original loss and an implicit regularizer. Then, we show that this implicit regularizer for (GD+M) is stronger than that of (GD) by factor of (frac1+β1-β), thus explaining why (GD+M) shows better generalization performance and higher test accuracy than (GD). Furthermore, we extend our analysis to the stochastic version of gradient descent with momentum (SGD+M) and characterize the continuous trajectory of the update of (SGD+M) in a pointwise sense. We explore the implicit regularization in (SGD+M) and (GD+M) through a series of experiments validating our theory.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

24 of 33 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Wide Residual Networkscited by 3,057
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  9. 2017Implicit Regularization in Matrix Factorizationcited by 318
  10. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  11. 2020Implicit Gradient Regularizationcited by 145
  12. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
12 more
  1. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  2. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  3. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  4. 2017Implicit Regularization in Deep Learningcited by 90
  5. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  6. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  7. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
  8. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  9. 2020Explicit Regularisation in Gaussian Noise Injectionscited by 37
  10. 2019Continuous Time Analysis of Momentum Methodscited by 32
  11. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 25
  12. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 8; 2025: 5; 2026: 5.

25 papers.

Sort:

By area:

  1. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47
  2. 2023On the Implicit Bias of Adamcited by 25
  3. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  4. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
  5. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 8
  6. 2023Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Boundscited by 6
  7. 2024Where Do Large Learning Rates Lead Us?cited by 6
  8. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
  9. 2023Deep Fusion: Efficient Network Training via Pre-trained Initializationscited by 4
  10. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4
  11. 2024The Impact of Geometric Complexity on Neural Collapse in Transfer Learningcited by 4
  12. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
13 more
  1. 2025Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Gamescited by 3
  2. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2
  3. 2024A Margin-based Multiclass Generalization Bound via Geometric Complexitycited by 2
  4. 2024Multiple Instance Verificationcited by 2
  5. 2023Acceleration and Implicit Regularization in Gaussian Phase Retrievalcited by 1
  6. 2025Learning by solving differential equationscited by 1
  7. 2025Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorationscited by 1
  8. 2026Implicit Regularization of Mini-Batch Training in Graph Neural Networkscited by 1
  9. 2024Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Trainingcited by 0
  10. 2024Corridor Geometry in Gradient-Based Optimizationcited by 0
  11. 2025Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysiscited by 0
  12. 2026Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approachcited by 0
  13. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0