mlsanityData to 7 Oct 2026

Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy

arXiv 2106.13673 ↗First posted 25 Jun 2021cs.LG cs.CR cs.DCcited by 66

Not on the map. Why?

Abstract (arXiv)

Providing privacy protection has been one of the primary motivations of Federated Learning (FL). Recently, there has been a line of work on incorporating the formal privacy notion of differential privacy with FL. To guarantee the client-level differential privacy in FL algorithms, the clients' transmitted model updates have to be clipped before adding privacy noise. Such clipping operation is substantially different from its counterpart of gradient clipping in the centralized differentially private SGD and has not been well-understood. In this paper, we first empirically demonstrate that the clipped FedAvg can perform surprisingly well even with substantial data heterogeneity when training neural networks, which is partly because the clients' updates become similar for several popular deep architectures. Based on this key observation, we provide the convergence analysis of a differential private (DP) FedAvg algorithm and highlight the relationship between clipping bias and the distribution of the clients' updates. To the best of our knowledge, this is the first work that rigorously investigates theoretical and empirical issues regarding the clipping operation in FL algorithms.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 25 Jun 2021.

Builds on

25 of 41 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  5. 2020Adaptive Federated Optimizationcited by 1,036
  6. 2019Deep Leakage from Gradientscited by 1,004
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  8. 2018LEAF: A Benchmark for Federated Settingscited by 780
  9. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  10. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 528
  11. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  12. 2020iDLG: Improved Deep Leakage from Gradientscited by 367
13 more
  1. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  2. 2018A Hybrid Approach to Privacy-Preserving Federated Learningcited by 208
  3. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  4. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  5. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  6. 2020LDP-Fed: Federated Learning with Local Differential Privacycited by 129
  7. 2019First Analysis of Local GD on Heterogeneous Datacited by 80
  8. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  9. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  10. 2020A Framework for Evaluating Gradient Leakage Attacks in Federated Learningcited by 63
  11. 2019Federated Learning with Bayesian Differential Privacycited by 60
  12. 2019Differential Privacy-enabled Federated Learning for Sensitive Health Datacited by 44
  13. 2020Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descentcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 24; 2024: 9; 2025: 14; 2026: 10.

66 papers.

Sort:

By area:

  1. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  2. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  3. 2022Mixed Differential Privacy in Computer Visioncited by 53
  4. 2023Recent Advances on Federated Learning: A Systematic Surveycited by 40
  5. 2022Differentially Private Federated Learning with Local Regularization and Sparsificationcited by 35
  6. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 35
  7. 2023LOKI: Large-scale Data Reconstruction Attack against Federated Learning through Model Manipulationcited by 35
  8. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  9. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19
  10. 2023Practical Differentially Private and Byzantine-resilient Federated Learningcited by 18
  11. 2023z-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learningcited by 17
  12. 2023Differentially Private SGD Without Clipping Bias: An Error-Feedback Approachcited by 13
38 more
  1. 2023Federated Learning over Coupled Graphscited by 12
  2. 2022Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learningcited by 10
  3. 2023Clip21: Error Feedback for Gradient Clippingcited by 10
  4. 2024Adaptive Gradient Clipping for Robust Federated Learningcited by 10
  5. 2023Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compressioncited by 9
  6. 2023ULDP-FL: Federated Learning with Across Silo User-Level Differential Privacycited by 9
  7. 2025DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Normcited by 9
  8. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  9. 2024Certifiably Byzantine-Robust Federated Conformal Predictioncited by 8
  10. 2022QUIC-FL: Quick Unbiased Compression for Federated Learningcited by 7
  11. 2023EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Datacited by 7
  12. 2023High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noisecited by 7
  13. 2025DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimationcited by 6
  14. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 4
  15. 2025Biased Federated Learning under Wireless Heterogeneitycited by 4
  16. 2023Coupling public and private gradient provably helps optimizationcited by 3
  17. 2023PriPrune: Quantifying and Preserving Privacy in Pruned Federated Learningcited by 3
  18. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  19. 2025Sharp High-Probability Rates for Nonlinear SGD under Heavy-Tailed Noise via Symmetrizationcited by 3
  20. 2022A Differentially Private Probabilistic Framework for Modeling the Variability Across Federated Datasets of Heterogeneous Multi-View Observationscited by 2
  21. 2022Local Model Reconstruction Attacks in Federated Learning and their Usescited by 2
  22. 2023U-Clip: On-Average Unbiased Stochastic Gradient Clippingcited by 2
  23. 2023Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification in the Presence of Data Heterogeneitycited by 2
  24. 2023PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advancecited by 2
  25. 2024TernaryVote: Differentially Private, Communication Efficient, and Byzantine Resilient Distributed Optimization on Heterogeneous Datacited by 2
  26. 2023Over-the-Air Federated Averaging with Limited Power and Privacy Budgetscited by 1
  27. 2023Differentially Private Wireless Federated Learning Using Orthogonal Sequencescited by 1
  28. 2023A Theoretical Analysis of Efficiency Constrained Utility-Privacy Bi-Objective Optimization in Federated Learningcited by 1
  29. 2024Private and Communication-Efficient Federated Learning based on Differentially Private Sketchescited by 1
  30. 2024Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guaranteescited by 1
  31. 2025Accelerating Differentially Private Federated Learning via Adaptive Extrapolationcited by 1
  32. 2026Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimizationcited by 1
  33. 2026Robust and Fast Training via Per-Sample Clippingcited by 1
  34. 2023Fair Differentially Private Federated Learning Frameworkcited by 0
  35. 2023Differentially Private Distributed Estimation and Learningcited by 0
  36. 2023Using Decentralized Aggregation for Federated Learning with Differential Privacycited by 0
  37. 2024Differentially-Private Multi-Tier Federated Learningcited by 0
  38. 2025Controlled privacy leakage propagation throughout overlapping grouped learningcited by 0

50 most cited of 66 All 66 (JSON)