mlsanityData to 14 Oct 2026

Clip21: Error Feedback for Gradient Clipping

arXiv 2305.18929 ↗First posted 30 May 2023cs.LG math.OC stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

Motivated by the increasing popularity and importance of large-scale training under differential privacy (DP) constraints, we study distributed gradient methods with gradient clipping, i.e., clipping applied to the gradients computed from local information at the nodes. While gradient clipping is an essential tool for injecting formal DP guarantees into gradient-based methods [1], it also induces bias which causes serious convergence issues specific to the distributed setting. Inspired by recent progress in the error-feedback literature which is focused on taming the bias/error introduced by communication compression operators such as Top-k [2], and mathematical similarities between the clipping operator and contractive compression operators, we design Clip21 -- the first provably effective and practically useful error feedback mechanism for distributed methods with gradient clipping. We prove that our method converges at the same Oleft(frac1Kright) rate as distributed gradient descent in the smooth nonconvex regime, which improves the previous best Oleft(frac1√Kright) rate which was obtained under significantly stronger assumptions. Our method converges significantly faster in practice than competing methods.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

27 of 38 references.

References · 27

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2018Sparsified SGD with Memorycited by 376
  4. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  5. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  6. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  7. 2018The Convergence of Sparsified Gradient Methodscited by 229
  8. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
  9. 2020Learning from History for Byzantine Robust Optimizationcited by 137
  10. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
  11. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 119
  12. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
15 more
  1. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 99
  2. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  3. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  4. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  5. 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 69
  6. 2020Linearly Converging Error Compensated SGDcited by 61
  7. 2020Error Compensated Distributed SGD Can Be Acceleratedcited by 45
  8. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  9. 2019Algorithms of Robust Stochastic Optimization Based on Mirror Descent Methodcited by 32
  10. 20223PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregationcited by 27
  11. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  12. 2021High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noisecited by 19
  13. 2019The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communicationcited by 14
  14. 2019Compressed Gradient Methods with Hessian-Aided Error Compensationcited by 6
  15. 2022The Bounded Gaussian Mechanism for Differential Privacycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 4; 2026: 1.

11 papers.

  1. 2024Adaptive Gradient Clipping for Robust Federated Learningcited by 11
  2. 2024Loss Landscape Characterization of Neural Networks without Over-Parametrizationcited by 8
  3. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  4. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  5. 2024Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Clipping Bias and Remediescited by 5
  6. 2024Towards Faster Decentralized Stochastic Optimization with Communication Compressioncited by 5
  7. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 5
  8. 2025Smoothed Normalization for Efficient Distributed Private Optimizationcited by 3
  9. 2025First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptionscited by 1
  10. 2026Byzantine-Robust and Differentially Private Federated Optimization under Weaker Assumptionscited by 1
  11. 2024Tracking the Median of Gradients with a Stochastic Proximal Point Methodcited by 0