mlsanityData to 7 Oct 2026

Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach

arXiv 2311.14632 ↗First posted 24 Nov 2023cs.LG cs.CRcited by 13

Not on the map. Why?

Abstract (arXiv)

Differentially Private Stochastic Gradient Descent with Gradient Clipping (DPSGD-GC) is a powerful tool for training deep learning models using sensitive data, providing both a solid theoretical privacy guarantee and high efficiency. However, using DPSGD-GC to ensure Differential Privacy (DP) comes at the cost of model performance degradation due to DP noise injection and gradient clipping. Existing research has extensively analyzed the theoretical convergence of DPSGD-GC, and has shown that it only converges when using large clipping thresholds that are dependent on problem-specific parameters. Unfortunately, these parameters are often unknown in practice, making it hard to choose the optimal clipping threshold. Therefore, in practice, DPSGD-GC suffers from degraded performance due to the it constant bias introduced by the clipping. In our work, we propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC, which not only offers a diminishing utility bound without inducing a constant clipping bias, but more importantly, it allows for an arbitrary choice of clipping threshold that is independent of the problem. We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP. Additionally, we demonstrate that under mild conditions, our algorithm can achieve nearly the same utility bound as DPSGD without gradient clipping. Our empirical results on Cifar-10/100 and E2E datasets, show that the proposed algorithm achieves higher accuracies than DPSGD while maintaining the same level of DP guarantee.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 24 Nov 2023.

Builds on

31 of 53 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019Deep Leakage from Gradientscited by 1,004
  6. 2017Renyi Differential Privacycited by 643
  7. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  8. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  9. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  10. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  11. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  12. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
19 more
  1. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 152
  2. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  3. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 131
  4. 2018Privacy Amplification by Iterationcited by 97
  5. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 88
  6. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  7. 2018A General Approach to Adding Differential Privacy to Iterative Training Procedurescited by 84
  8. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  9. 2018Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learningcited by 75
  10. 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 66
  11. 2020On Differentially Private Stochastic Convex Optimization with Heavy-tailed Datacited by 51
  12. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  13. 2021Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Datacited by 47
  14. 2022Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)cited by 45
  15. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 35
  16. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  17. 2021Fast and Memory Efficient Differentially Private-SGD via JL Projectionscited by 29
  18. 2020Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descentcited by 27
  19. 2021On the Convergence of Differentially Private Federated Learning on Non-Lipschitz Objectives, and with Normalized Client Updatescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 34
  2. 2024DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reductioncited by 10
  3. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  4. 2024Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusioncited by 4
  5. 2024Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Clipping Bias and Remediescited by 4
  6. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 4
  7. 2025On Optimal Hyperparameters for Differentially Private Deep Transfer Learningcited by 1
  8. 2026FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correctioncited by 1
  9. 2026Efficient DP-SGD for LLMs with Randomized Clippingcited by 1
  10. 2025Balancing Utility and Privacy: Dynamically Private SGD with Random Projectioncited by 0
  11. 2026Differential Privacy Image Generation with Reconstruction Loss and Noise Injection Using an Error Feedback SGDcited by 0
  12. 2026Lap2: Revisiting Laplace DP-SGD for High Dimensions via Majorization Theorycited by 0
1 more
  1. 2026PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantizationcited by 0