mlsanityData to 14 Oct 2026

Why gradient clipping accelerates training: A theoretical justification for adaptivity

arXiv 1905.11881 ↗First posted 28 May 2019math.OC cs.LGcited by 341

Not on the map. Why?

Abstract (arXiv)

We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that gradient smoothness, a concept central to the analysis of first-order optimization algorithms that is often assumed to be a constant, demonstrates significant variability along the training trajectory of deep neural networks. Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient. These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a fixed bound on smoothness. These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used Lipschitz smoothness assumption. Under the new condition, we prove that two popular methods, namely, gradient clipping and normalized gradient, converge arbitrarily faster than gradient descent with fixed stepsize. We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.

Key facts. Cited by 341 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 35; 2022: 40; 2023: 61; 2024: 84; 2025: 65; 2026: 56.

341 papers.

Sort:

By area:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  2. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  3. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  4. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  5. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  6. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  7. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  8. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  9. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 94
  10. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  11. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  12. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
38 more
  1. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  2. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  3. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  4. 2024Scalable Optimization in the Modular Normcited by 68
  5. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  6. 2022Reachability Constrained Reinforcement Learningcited by 59
  7. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 57
  8. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 53
  9. 2021Quantifying Uncertainty in Deep Spatiotemporal Forecastingcited by 48
  10. 2022Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)cited by 48
  11. 2024SkateFormer: Skeletal-Temporal Transformer for Human Action Recognitioncited by 47
  12. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  13. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  14. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 43
  15. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  16. 2023Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGDcited by 41
  17. 2022Mitigating Memorization of Noisy Labels by Clipping the Model Predictioncited by 40
  18. 2025In Search of Adam's Secret Saucecited by 40
  19. 2021Non-convex Distributionally Robust Optimization: Non-asymptotic Analysiscited by 39
  20. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  21. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 36
  22. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  23. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 31
  24. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  25. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  26. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  27. 2023Parameter-Agnostic Optimization under Relaxed Smoothnesscited by 29
  28. 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 28
  29. 2022Multitrack Music Transformercited by 28
  30. 2021Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learningcited by 27
  31. 2022Robustifying Vision Transformer without Retraining from Scratch by Test-Time Class-Conditional Feature Alignmentcited by 27
  32. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  33. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  34. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 26
  35. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 25
  36. 2022CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videoscited by 23
  37. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  38. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 23

50 most cited of 341 All 341 (JSON)