mlsanityData to 7 Oct 2026

Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness

arXiv 2102.06489 ↗First posted 12 Feb 2021math.OC cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient algorithms are often unstable when applied to functions that do not have Lipschitz-continuous and/or bounded gradients. Gradient clipping is a simple and effective technique to stabilize the training process for problems that are prone to the exploding gradient problem. Despite its widespread popularity, the convergence properties of the gradient clipping heuristic are poorly understood, especially for stochastic problems. This paper establishes both qualitative and quantitative convergence results of the clipped stochastic (sub)gradient method (SGD) for non-smooth convex functions with rapidly growing subgradients. Our analyses show that clipping enhances the stability of SGD and that the clipped SGD algorithm enjoys finite convergence rates in many cases. We also study the convergence of a clipped method with momentum, which includes clipped SGD as a special case, for weakly convex problems under standard assumptions. With a novel Lyapunov analysis, we show that the proposed method achieves the best-known rate for the considered class of problems, demonstrating the effectiveness of clipped methods also in this regime. Numerical results confirm our theoretical developments.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 17 of its 44 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

17 of 44 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  3. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  4. 2018Stochastic model-based minimization of weakly convex functionscited by 157
  5. 2020Momentum Improves Normalized SGDcited by 136
  6. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  7. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 88
  8. 2015Beyond Convexity: Stochastic Quasi-Convex Optimizationcited by 75
  9. 2018Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivitycited by 61
  10. 2018A Single Time-Scale Stochastic Approximation Method for Nested Stochastic Optimizationcited by 53
  11. 2017Stochastic Methods for Composite and Weakly Convex Optimization Problemscited by 48
  12. 2019The importance of better models in stochastic optimizationcited by 32
5 more
  1. 2020Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimizationcited by 32
  2. 2017Proximally Guided Stochastic Subgradient Method for Nonsmooth, Nonconvex Problemscited by 30
  3. 2019Stochastic algorithms with geometric step decay converge linearly on sharp functionscited by 14
  4. 2017A Stochastic Trust Region Algorithm Based on Careful Step Normalizationcited by 10
  5. 2020Convergence of adaptive algorithms for weakly convex constrained optimizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 12; 2024: 8; 2025: 6; 2026: 7.

37 papers.

Sort:

By area:

  1. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  2. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  3. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  4. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
  5. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  6. 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
  7. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  8. 2022Clipped Stochastic Methods for Variational Inequalities with Heavy-Tailed Noisecited by 13
  9. 2023Clip21: Error Feedback for Gradient Clippingcited by 10
  10. 2023Understanding Fairness Surrogate Functions in Algorithmic Fairnesscited by 8
  11. 2024Contextual Bandits for Unbounded Context Distributionscited by 8
  12. 2023EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Datacited by 7
25 more
  1. 2023High Probability Analysis for Non-Convex Stochastic Optimization with Clippingcited by 5
  2. 2022CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPUcited by 4
  3. 2023Rethinking SIGN Training: Provable Nonconvex Acceleration without First- and Second-Order Gradient Lipschitzcited by 4
  4. 2024Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Clipping Bias and Remediescited by 4
  5. 2024Error Feedback under (L0,L1)-Smoothness: Normalization and Momentumcited by 4
  6. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3
  7. 2024Stochastic Weakly Convex Optimization Beyond Lipschitz Continuitycited by 3
  8. 2023U-Clip: On-Average Unbiased Stochastic Gradient Clippingcited by 2
  9. 2024Parallel Momentum Methods Under Biased Gradient Estimationscited by 2
  10. 2025On Efficiency-Effectiveness Trade-off of Diffusion-based Recommenderscited by 2
  11. 2024Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentumcited by 1
  12. 2024Private and Communication-Efficient Federated Learning based on Differentially Private Sketchescited by 1
  13. 2025Accelerated Gradient Methods with Biased Gradient Estimates: Risk Sensitivity, High-Probability Guarantees, and Large Deviation Boundscited by 1
  14. 2025Gradient Shaping Beyond Clipping: A Functional Perspective on Update Magnitude Controlcited by 1
  15. 2025Stabilizing Multimodal Autoencoders: A Theoretical and Empirical Analysis of Fusion Strategiescited by 1
  16. 2026How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNscited by 1
  17. 2023Unified analysis of SGD-type methodscited by 0
  18. 2023Adaptive Strategies in Non-convex Optimizationcited by 0
  19. 2025Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learningcited by 0
  20. 2026Stability and Generalization of Stochastic Nonconvex Optimization under Heavy-Tailed Noise: From Gradient Norm to Goldstein Stationaritycited by 0
  21. 2026Understanding vision transformer robustness through the lens of out-of-distribution detectioncited by 0
  22. 2026Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalizationcited by 0
  23. 2026Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglerscited by 0
  24. 2026On the Computational and Statistical Efficiency of the Empirical Maximum Entropy on the Mean Methodcited by 0
  25. 2026Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothnesscited by 0