mlsanityData to 14 Oct 2026

On the accuracy and efficiency of group-wise clipping in differentially private optimization

arXiv 2310.19215 ↗First posted 30 Oct 2023cs.LG cs.CC cs.CRcited by 7

Not on the map. Why?

Abstract (arXiv)

Recent advances have substantially improved the accuracy, memory cost, and training speed of differentially private (DP) deep learning, especially on large vision and language models with millions to billions of parameters. In this work, we thoroughly study the per-sample gradient clipping style, a key component in DP optimization. We show that different clipping styles have the same time complexity but instantiate an accuracy-memory trade-off: while the all-layer clipping (of coarse granularity) is the most prevalent and usually gives the best accuracy, it incurs heavier memory cost compared to other group-wise clipping, such as the layer-wise clipping (of finer granularity). We formalize this trade-off through our convergence theory and complexity analysis. Importantly, we demonstrate that the accuracy gap between group-wise clipping and all-layer clipping becomes smaller for larger models, while the memory advantage of the group-wise clipping remains. Consequently, the group-wise clipping allows DP optimization of large models to achieve high accuracy and low peak memory simultaneously.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

29 of 37 references.

References · 29

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  6. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  7. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  8. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  9. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  10. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  11. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  12. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
17 more
  1. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  2. 2021Numerical Composition of Differential Privacycited by 180
  3. 2019Deep Learning with Gaussian Differential Privacycited by 114
  4. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
  5. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  6. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 88
  7. 2021Optimal Accounting of Differential Privacy via Characteristic Functioncited by 86
  8. 2018A General Approach to Adding Differential Privacy to Iterative Training Procedurescited by 84
  9. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  10. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  11. 2022Mixed Differential Privacy in Computer Visioncited by 54
  12. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  13. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 48
  14. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 48
  15. 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 47
  16. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 41
  17. 2021On the Convergence and Calibration of Deep Learning with Differential Privacycited by 36

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 4.

7 papers.

  1. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 41
  2. 2025Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Trainingcited by 15
  3. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  4. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  5. 2024Can large language models be privacy preserving and fair medical coders?cited by 1
  6. 2025FlashDP: Private Training Large Language Models with Efficient DP-SGDcited by 1
  7. 2025Towards Robust Fact-Checking: A Multi-Agent System with Advanced Evidence Retrievalcited by 0