mlsanityData to 14 Oct 2026

Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees

arXiv 2305.01588 ↗First posted 2 May 2023cs.LG cs.DC math.OCcited by 83

Not on the map. Why?

Abstract (arXiv)

Gradient clipping is a popular modification to standard (stochastic) gradient descent, at every iteration limiting the gradient norm to a certain value c >0. It is widely used for example for stabilizing the training of deep learning models (Goodfellow et al., 2016), or for enforcing differential privacy (Abadi et al., 2016). Despite popularity and simplicity of the clipping mechanism, its convergence guarantees often require specific values of c and strong noise assumptions. In this paper, we give convergence guarantees that show precise dependence on arbitrary clipping thresholds c and show that our guarantees are tight with both deterministic and stochastic gradients. In particular, we show that (i) for deterministic gradient descent, the clipping threshold only affects the higher-order terms of convergence, (ii) in the stochastic setting convergence to the true optimum cannot be guaranteed under the standard noise assumption, even under arbitrary small step-sizes. We give matching upper and lower bounds for convergence of the gradient norm when running clipped SGD, and illustrate these results with experiments.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 2 May 2023.

Builds on

21 of 40 references.

References · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2016Deep Learning with Differential Privacycited by 2,728
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  5. 2019On the Convergence of Adam and Beyondcited by 841
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  7. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  8. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  9. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 186
  10. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
  11. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 128
  12. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
9 more
  1. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  2. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  3. 2015Beyond Convexity: Stochastic Quasi-Convex Optimizationcited by 76
  4. 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 69
  5. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  6. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 36
  7. 2016The Power of Normalization: Faster Evasion of Saddle Pointscited by 29
  8. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  9. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 20; 2025: 26; 2026: 29.

83 papers.

Sort:

By area:

  1. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 49
  2. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  3. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  4. 2023Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clippingcited by 17
  5. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  6. 2023Differentially Private SGD Without Clipping Bias: An Error-Feedback Approachcited by 14
  7. 2025Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under (L0, L1)-Smoothnesscited by 13
  8. 2024Taming Nonconvex Stochastic Mirror Descent with General Bregman Divergencecited by 11
  9. 2024Adaptive Gradient Clipping for Robust Federated Learningcited by 11
  10. 2026Enhancing LLM Training via Spectral Clippingcited by 10
  11. 2023Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compressioncited by 9
  12. 2023On the Tradeoff between Privacy Preservation and Byzantine-Robustness in Decentralized Learningcited by 8
38 more
  1. 2024To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensionscited by 7
  2. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  3. 2023Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimizationcited by 6
  4. 2024Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Clipping Bias and Remediescited by 5
  5. 2024BadGD: A unified data-centric framework to identify gradient descent vulnerabilitiescited by 5
  6. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 5
  7. 2024A New First-Order Meta-Learning Algorithm with Convergence Guaranteescited by 4
  8. 2024Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimizationcited by 4
  9. 2024Error Feedback under (L0,L1)-Smoothness: Normalization and Momentumcited by 4
  10. 2025An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Lossescited by 4
  11. 2025On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approachcited by 4
  12. 2024Stochastic Weakly Convex Optimization Beyond Lipschitz Continuitycited by 3
  13. 2024Parameter-free Clipped Gradient Descent Meets Polyakcited by 3
  14. 2025Smoothed Normalization for Efficient Distributed Private Optimizationcited by 3
  15. 2025Armijo Line-search Can Make (Stochastic) Gradient Descent Provably Fastercited by 3
  16. 2025Gradient-Normalized Smoothness for Optimization with Approximate Hessianscited by 3
  17. 2024Private and Communication-Efficient Federated Learning based on Differentially Private Sketchescited by 2
  18. 2025Statistical Inference for Differentially Private Stochastic Gradient Descentcited by 2
  19. 2025Unifying Agent Interaction and World Information for Multi-agent Coordinationcited by 2
  20. 2026Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noisecited by 2
  21. 2026Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspectivecited by 2
  22. 2026How to find expressible and trainable parameterized quantum circuits?cited by 2
  23. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  24. 2023The Relative Gaussian Mechanism and its Application to Private Gradient Descentcited by 1
  25. 2024Variance Control for Black Box Variational Inference Using The James-Stein Estimatorcited by 1
  26. 2024Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Ratescited by 1
  27. 2024Noise is All You Need: Private Second-Order Convergence of Noisy SGDcited by 1
  28. 2024Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysiscited by 1
  29. 2025Private Rate-Constrained Optimization with Applications to Fair Learningcited by 1
  30. 2025Near-Optimal Convergence of Accelerated Gradient Methods under Generalized and (L0, L1)-Smoothnesscited by 1
  31. 2025Accelerated Gradient Methods with Biased Gradient Estimates: Risk Sensitivity, High-Probability Guarantees, and Large Deviation Boundscited by 1
  32. 2025Gradient Shaping Beyond Clipping: A Functional Perspective on Update Magnitude Controlcited by 1
  33. 2025High-Probability Convergence Guarantees of Decentralized SGDcited by 1
  34. 2025On Optimal Hyperparameters for Differentially Private Deep Transfer Learningcited by 1
  35. 2025First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptionscited by 1
  36. 2026Deterministic Denominator Design for Localized Tamed Stochastic Gradient Langevin Dynamicscited by 1
  37. 2023A Single-Loop Deep Actor-Critic Algorithm for Constrained Reinforcement Learning with Provable Convergencecited by 0
  38. 2024Tracking the Median of Gradients with a Stochastic Proximal Point Methodcited by 0

50 most cited of 83 All 83 (JSON)