mlsanityData to 7 Oct 2026

Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger

arXiv 2206.07136 ↗First posted 14 Jun 2022cs.LG cs.CL cs.CRcited by 87

Not on the map. Why?

Abstract (arXiv)

Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping threshold R, however, is vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called automatic clipping, that eliminates the need to tune R for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB and many others. The automatic variants are as private and computationally efficient as existing DP optimizers, but require no DP-specific hyperparameters and thus make DP training as amenable as the standard non-private training. We give a rigorous convergence analysis of automatic DP-SGD in the non-convex setting, showing that it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric gradient noise assumption of the per-sample gradients (commonly used in the non-DP literature). We demonstrate on various language and vision tasks that automatic clipping outperforms or matches the state-of-the-art, and can be easily employed with minimal changes to existing codebases.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 52 of its 92 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

52 of 92 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  11. 2016Deep Learning with Differential Privacycited by 2,682
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
38 more
  1. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  5. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  8. 2017Renyi Differential Privacycited by 643
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  11. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  12. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  13. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  14. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  15. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  16. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  17. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  18. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  19. 2021Numerical Composition of Differential Privacycited by 174
  20. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  21. 2020Momentum Improves Normalized SGDcited by 136
  22. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 121
  23. 2019Deep Learning with Gaussian Differential Privacycited by 112
  24. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  25. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  26. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  27. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  28. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  29. 2021Optimal Accounting of Differential Privacy via Characteristic Functioncited by 83
  30. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  31. 2022Mixed Differential Privacy in Computer Visioncited by 53
  32. 2017Natasha 2: Faster Non-Convex Optimization Than SGDcited by 49
  33. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  34. 2021On the Convergence and Calibration of Deep Learning with Differential Privacycited by 36
  35. 2021Fast and Memory Efficient Differentially Private-SGD via JL Projectionscited by 29
  36. 2017Revisiting Normalized Gradient Descent: Fast Evasion of Saddle Pointscited by 28
  37. 2022Differentially private training of residual networks with scale normalisationcited by 26
  38. 2020Stochastic Normalized Gradient Descent with Momentum for Large-Batch Trainingcited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 18; 2024: 20; 2025: 28; 2026: 15.

87 papers.

Sort:

By area:

  1. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 85
  2. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  3. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  4. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  5. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  6. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  7. 2022Differentially Private Learning with Per-Sample Adaptive Clippingcited by 21
  8. 2023Practical Differentially Private and Byzantine-resilient Federated Learningcited by 18
  9. 2024Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuningcited by 14
  10. 2025Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Trainingcited by 14
  11. 2023Differentially Private SGD Without Clipping Bias: An Error-Feedback Approachcited by 13
  12. 2023DP-SGD Without Clipping: The Lipschitz Neural Network Waycited by 11
38 more
  1. 2023Information Flow Control in Machine Learning through Modular Model Architecturecited by 11
  2. 2024Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisationcited by 11
  3. 2024DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reductioncited by 10
  4. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  5. 2024DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reductioncited by 8
  6. 2022Differentially Private Image Classification from Featurescited by 7
  7. 2023Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Trainingcited by 7
  8. 2023On the accuracy and efficiency of group-wise clipping in differentially private optimizationcited by 7
  9. 2023Zero redundancy distributed learning with differential privacycited by 7
  10. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  11. 2022Differentially Private Optimizers Can Learn Adversarially Robust Modelscited by 6
  12. 2023Augment then Smooth: Reconciling Differential Privacy with Certified Robustnesscited by 6
  13. 2023PLAN: Variance-Aware Private Mean Estimationcited by 6
  14. 2025DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimationcited by 6
  15. 2024Delving into Differentially Private Transformercited by 5
  16. 2023Tractable MCMC for Private Learning with Pure and Gaussian Differential Privacycited by 4
  17. 2024Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposurecited by 4
  18. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 4
  19. 2024DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learningcited by 4
  20. 2025Crowding Out The Noise: Algorithmic Collective Action Under Differential Privacycited by 4
  21. 2023Coupling public and private gradient provably helps optimizationcited by 3
  22. 2024Defending Against Data Reconstruction Attacks in Federated Learning: An Information Theory Approachcited by 3
  23. 2024PFGuard: A Generative Framework with Privacy and Fairness Safeguardscited by 3
  24. 2023DPFormer: Learning Differentially Private Transformer on Long-Tailed Datacited by 2
  25. 2023A Comprehensive Study of Privacy Risks in Curriculum Learningcited by 2
  26. 2023PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advancecited by 2
  27. 2025Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clippingcited by 2
  28. 2025Evading Data Provenance in Deep Neural Networkscited by 2
  29. 2025Understanding Private Learning From Feature Perspectivecited by 2
  30. 2022SA-DPSGD: Differentially Private Stochastic Gradient Descent based on Simulated Annealingcited by 1
  31. 2023ALI-DPFL: Differentially Private Federated Learning with Adaptive Local Iterationscited by 1
  32. 2023Does Differential Privacy Prevent Backdoor Attacks in Practice?cited by 1
  33. 2024Federated Learning for Heterogeneous Electronic Health Record Systems with Cost Effective Participant Selectioncited by 1
  34. 2024Noise is All You Need: Private Second-Order Convergence of Noisy SGDcited by 1
  35. 2024Privately Learning from Graphs with Applications in Fine-tuning Large Language Modelscited by 1
  36. 2024Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weightcited by 1
  37. 2025Synergizing Large Language Models and Task-specific Models for Time Series Anomaly Detectioncited by 1
  38. 2025Smoothed Normalization for Efficient Distributed Private Optimizationcited by 1

50 most cited of 87 All 87 (JSON)