mlsanityData to 14 Oct 2026

Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization

arXiv 2010.09063 ↗First posted 18 Oct 2020cs.LG cs.CR cs.PFcited by 58

Not on the map. Why?

Abstract (arXiv)

A common pain point in differentially private machine learning is the significant runtime overhead incurred when executing Differentially Private Stochastic Gradient Descent (DPSGD), which may be as large as two orders of magnitude. We thoroughly demonstrate that by exploiting powerful language primitives, including vectorization, just-in-time compilation, and static graph optimization, one can dramatically reduce these overheads, in many cases nearly matching the best non-private running times. These gains are realized in two frameworks: JAX and TensorFlow. JAX provides rich support for these primitives as core features of the language through the XLA compiler. We also rebuild core parts of TensorFlow Privacy, integrating features from TensorFlow 2 as well as XLA compilation, granting significant memory and runtime improvements over the current release version. These approaches allow us to achieve up to 50x speedups in comparison to the best alternatives. Our code is available at https://github.com/TheSalon/fast-dpsgd.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 19; 2023: 10; 2024: 9; 2025: 4; 2026: 2.

58 papers.

Sort:

  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  2. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  3. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  4. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  5. 2021Large-Scale Differentially Private BERTcited by 118
  6. 2021Deep Learning with Label Differential Privacycited by 117
  7. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  8. 2021PPFL: Privacy-preserving Federated Learning with Trusted Execution Environmentscited by 93
  9. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  10. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  11. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  12. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 48
38 more
  1. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 48
  2. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 41
  3. 2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
  4. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  5. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 30
  6. 2021Fast and Memory Efficient Differentially Private-SGD via JL Projectionscited by 29
  7. 2022Differentially private training of residual networks with scale normalisationcited by 26
  8. 2023Selective Pre-training for Private Fine-tuningcited by 23
  9. 2023No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device MLcited by 21
  10. 2021DPNAS: Neural Architecture Search for Deep Learning with Differential Privacycited by 20
  11. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  12. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 16
  13. 2022DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Samplingcited by 13
  14. 2022Privacy-Preserving Domain Adaptation of Semantic Parserscited by 12
  15. 2022Privacy-Preserving Models for Legal Natural Language Processingcited by 10
  16. 2023Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Modelscited by 10
  17. 2022SF-PATE: Scalable, Fair, and Private Aggregation of Teacher Ensemblescited by 8
  18. 2021DP-FP: Differentially Private Forward Propagation for Large Modelscited by 7
  19. 2022Differentially Private Image Classification from Featurescited by 7
  20. 2023On the accuracy and efficiency of group-wise clipping in differentially private optimizationcited by 7
  21. 2023Zero redundancy distributed learning with differential privacycited by 7
  22. 2023DP-NMT: Scalable Differentially-Private Machine Translationcited by 7
  23. 2022Private Ad Modeling with DP-SGDcited by 6
  24. 2024Delving into Differentially Private Transformercited by 5
  25. 2022DiVa: An Accelerator for Differentially Private Machine Learningcited by 4
  26. 2023Generative Model-Based Attack on Learnable Image Encryption for Privacy-Preserving Deep Learningcited by 4
  27. 2024Efficient and Scalable Implementation of Differentially Private Deep Learning without Shortcutscited by 3
  28. 2024TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers have Pre-Trained Modelscited by 3
  29. 2021D3p -- A Python Package for Differentially-Private Probabilistic Programmingcited by 2
  30. 2023DPFormer: Learning Differentially Private Transformer on Long-Tailed Datacited by 2
  31. 2024Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clippingcited by 2
  32. 2024Differentially Private Prototypes for Imbalanced Transfer Learningcited by 2
  33. 2024Training Large ASR Encoders with Differential Privacycited by 2
  34. 2025Private Zeroth-Order Optimization with Public Datacited by 2
  35. 2021Towards Prior-Free Approximately Truthful One-Shot Auction Learning via Differential Privacycited by 1
  36. 2022Rieoptax: Riemannian Optimization in JAXcited by 1
  37. 2024Universally Harmonizing Differential Privacy Mechanisms for Federated Learning: Boosting Accuracy and Convergencecited by 1
  38. 2024Noise is All You Need: Private Second-Order Convergence of Noisy SGDcited by 1

50 most cited of 58 All 58 (JSON)