mlsanityData to 14 Oct 2026

Does Momentum Change the Implicit Regularization on Separable Data?

arXiv 2110.03891 ↗First posted 8 Oct 2021cs.LG math.OCcited by 22

Not on the map. Why?

Abstract (arXiv)

The momentum acceleration technique is widely adopted in many optimization algorithms. However, there is no theoretical answer on how the momentum affects the generalization performance of the optimization algorithms. This paper studies this problem by analyzing the implicit regularization of momentum-based optimization. We prove that on the linear classification problem with separable data and exponential-tailed loss, gradient descent with momentum (GDM) converges to the L2 max-margin solution, which is the same as vanilla gradient descent. That means gradient descent with momentum acceleration still converges to a low-complexity model, which guarantees their generalization. We then analyze the stochastic and adaptive variants of GDM (i.e., SGDM and deterministic Adam) and show they also converge to the L2 max-margin solution. Technically, to overcome the difficulty of the error accumulation in analyzing the momentum, we construct new potential functions to analyze the gap between the model parameter and the max-margin solution. Numerical experiments are conducted and support our theoretical results.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

37 of 52 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  9. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  11. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  12. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
25 more
  1. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  2. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  3. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  4. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  5. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  6. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  7. 2020Directional convergence and alignment in deep learningcited by 178
  8. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  9. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  10. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  11. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 119
  12. 2018Convergence of Gradient Descent on Separable Datacited by 110
  13. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
  14. 2014Global convergence of the Heavy-ball method for convex optimizationcited by 61
  15. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 59
  16. 2018A Unified Analysis of Stochastic Momentum Methods for Deep Learningcited by 55
  17. 2013Fast Convergence of Stochastic Gradient Descent under a Strong Growth Conditioncited by 54
  18. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  19. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
  20. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  21. 2013Margins, Shrinkage, and Boostingcited by 35
  22. 2021Fast Margin Maximization via Dual Accelerationcited by 33
  23. 2019The Implicit Bias of AdaGrad on Separable Datacited by 16
  24. 2018Non-ergodic Convergence Analysis of Heavy-Ball Algorithmscited by 14
  25. 2021The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methodscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 8; 2026: 6.

22 papers.

Sort:

  1. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 62
  2. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  3. 2024The Implicit Bias of Adam on Separable Datacited by 36
  4. 2023Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropycited by 30
  5. 2023On the Implicit Bias of Adamcited by 27
  6. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  7. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
  8. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 13
  9. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 10
  10. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
  11. 2025Towards Understanding Generalization in DP-GD: A Case Study in Training Two-Layer CNNscited by 4
  12. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
10 more
  1. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 3
  2. 2024A Mirror Descent Perspective of Smoothed Sign Descentcited by 3
  3. 2026OLion: Approaching the Hadamard Ideal by Intersecting Spectral and ℓ∞ Implicit Biasescited by 3
  4. 2023Faster Margin Maximization Rates for Generic and Adversarially Robust Optimization Methodscited by 0
  5. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  6. 2025Variational Deep Learning via Implicit Regularizationcited by 0
  7. 2025Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysiscited by 0
  8. 2026Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approachcited by 0
  9. 2026Optimizer-dependent training dynamics converge to the same one-third optimal data scalingcited by 0
  10. 2026From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Biascited by 0