Data to 7 Oct 2026

Momentum Improves Normalized SGD

arXiv 2002.03305 ↗First posted 9 Feb 2020cs.LG math.OC stat.MLcited by 136

Not on the map. Why?

Abstract (arXiv)

We provide an improved analysis of normalized SGD showing that adding momentum provably removes the need for large batch sizes on non-convex objectives. Then, we consider the case of objectives with bounded second derivative and show that in this case a small tweak to the momentum formula allows normalized SGD with momentum to find an ε-critical point in O(1/ε3.5) iterations, matching the best-known rates without accruing any logarithmic factors or dependence on dimension. We also provide an adaptive method that automatically improves convergence rates when the variance in the gradients is small. Finally, we show that our method is effective when employed on popular large scale tasks such as ResNet-50 and BERT pretraining, matching the performance of the disparate methods used to get state-of-the-art results on both tasks.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 10; 2023: 20; 2024: 29; 2025: 36; 2026: 31.

136 papers.

Sort:

By area:

2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
2023Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversioncited by 42

38 more

2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
2022Towards understanding how momentum improves generalization in deep learningcited by 40
2021Non-convex Distributionally Robust Optimization: Non-asymptotic Analysiscited by 39
2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 35
2024Does SGD really happen in tiny subspaces?cited by 35
2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
2023Momentum Provably Improves Error Feedback!cited by 31
2023Parameter-Agnostic Optimization under Relaxed Smoothnesscited by 29
2025On the Convergence of Muon and Beyondcited by 28
2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 22
2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
2022Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Datacited by 20
2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
2023Practical Differentially Private and Byzantine-resilient Federated Learningcited by 18
2024Random Scaling and Momentum for Non-smooth Non-convex Optimizationcited by 18
2024Revisiting Gradient Normalization and Clipping for Nonconvex SGD under Heavy-Tailed Noise: Necessity, Sufficiency, and Accelerationcited by 18
2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 17
2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
2023On the Role of Server Momentum in Federated Learningcited by 16
2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
2024Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysiscited by 15
2021Better SGD using Second-order Momentumcited by 14
2024Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variancecited by 14
2024On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyondcited by 13
2025Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under (L0, L1)-Smoothnesscited by 12
2022A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimizationcited by 11
2024Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPscited by 11
2025Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noisecited by 11
2022Momentum Aggregation for Private Non-convex ERMcited by 10
2024Non-convex Stochastic Composite Optimization with Polyak Momentumcited by 9
2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
2025Gradient Multi-Normalization for Stateless and Scalable LLM Trainingcited by 9
2024DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reductioncited by 8

50 most cited of 136 All 136 (JSON)