mlsanityData to 7 Oct 2026

Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clipping

arXiv 2009.03106 ↗First posted 7 Sep 2020cs.LG stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

Recent work on Renyi Differential Privacy has shown the feasibility of applying differential privacy to deep learning tasks. Despite their promise, however, differentially private deep networks often lag far behind their non-private counterparts in accuracy, showing the need for more research in model architectures, optimizers, etc. One of the barriers to this expanded research is the training time -- often orders of magnitude larger than training non-private networks. The reason for this slowdown is a crucial privacy-related step called "per-example gradient clipping" whose naive implementation undoes the benefits of batch training with GPUs. By analyzing the back-propagation equations we derive new methods for per-example gradient clipping that are compatible with auto-differentiation (e.g., in PyTorch and TensorFlow) and provide better GPU utilization. Our implementation in PyTorch showed significant training speed-ups (by factors of 54x - 94x for training various models with batch sizes of 128). These techniques work for a variety of architectural choices including convolutional layers, recurrent networks, attention, residual blocks, etc.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 7; 2023: 8; 2024: 7; 2025: 11; 2026: 7.

46 papers.

Sort:

  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  2. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  3. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  4. 2024Data Shapley in One Training Runcited by 76
  5. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  6. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  7. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  8. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  9. 2021Fast and Memory Efficient Differentially Private-SGD via JL Projectionscited by 29
  10. 2023Selective Pre-training for Private Fine-tuningcited by 22
  11. 2021DPNAS: Neural Architecture Search for Deep Learning with Differential Privacycited by 20
  12. 2024From Challenges and Pitfalls to Recommendations and Opportunities: Implementing Federated Learning in Healthcarecited by 16
34 more
  1. 2023DP-SGD Without Clipping: The Lipschitz Neural Network Waycited by 11
  2. 2023On the accuracy and efficiency of group-wise clipping in differentially private optimizationcited by 7
  3. 2023Zero redundancy distributed learning with differential privacycited by 7
  4. 2021DP-FP: Differentially Private Forward Propagation for Large Modelscited by 6
  5. 2022Private Ad Modeling with DP-SGDcited by 6
  6. 2024Differentially Private Representation Learning via Image Captioningcited by 5
  7. 2024Delving into Differentially Private Transformercited by 5
  8. 2023All Rivers Run to the Sea: Private Learning with Asymmetric Flowscited by 4
  9. 2022DiVa: An Accelerator for Differentially Private Machine Learningcited by 3
  10. 2021Masked LARk: Masked Learning, Aggregation and Reporting worKflowcited by 2
  11. 2023DPFormer: Learning Differentially Private Transformer on Long-Tailed Datacited by 2
  12. 2024Differentially Private Prototypes for Imbalanced Transfer Learningcited by 2
  13. 2025Learning with Differentially Private (Sliced) Wasserstein Gradientscited by 2
  14. 2025Memory-Efficient Differentially Private Training with Gradient Random Projectioncited by 2
  15. 2025Layer-Aware Influence for Online Data Valuation Estimationcited by 2
  16. 2026JAX-Privacy: A library for differentially private machine learningcited by 2
  17. 2022Rieoptax: Riemannian Optimization in JAXcited by 1
  18. 2024LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Modelscited by 1
  19. 2024Privately Learning from Graphs with Applications in Fine-tuning Large Language Modelscited by 1
  20. 2025Inclusive Federated Learning Through Compliance-Weighted Noise Allocation in Healthcare AIcited by 1
  21. 2025FlashDP: Private Training Large Language Models with Efficient DP-SGDcited by 1
  22. 2025DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Schedulingcited by 1
  23. 2025Privacy-Preserving Generation of Clinical Narratives from Medical Terminologiescited by 1
  24. 2025Correlating Cross-Iteration Noise for DP-SGD using Model Curvaturecited by 1
  25. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  26. 2026When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learningcited by 1
  27. 2026Robust and Fast Training via Per-Sample Clippingcited by 1
  28. 2026Efficient DP-SGD for LLMs with Randomized Clippingcited by 1
  29. 2026Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tailcited by 1
  30. 2023Differentially Private Optimization for Non-Decomposable Objective Functionscited by 0
  31. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  32. 2025Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniquenesscited by 0
  33. 2026Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectificationcited by 0
  34. 2026Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD?cited by 0