mlsanityData to 7 Oct 2026

Fast and Memory Efficient Differentially Private-SGD via JL Projections

arXiv 2102.03013 ↗First posted 5 Feb 2021cs.LG cs.CRcited by 29

Not on the map. Why?

Abstract (arXiv)

Differentially Private-SGD (DP-SGD) of Abadi et al. (2016) and its variations are the only known algorithms for private training of large scale neural networks. This algorithm requires computation of per-sample gradients norms which is extremely slow and memory intensive in practice. In this paper, we present a new framework to design differentially private optimizers called DP-SGD-JL and DP-Adam-JL. Our approach uses Johnson-Lindenstrauss (JL) projections to quickly approximate the per-sample gradient norms without exactly computing them, thus making the training time and memory requirements of our optimizers closer to that of their non-DP versions. Unlike previous attempts to make DP-SGD faster which work only on a subset of network architectures or use compiler techniques, we propose an algorithmic solution which works for any network in a black-box manner which is the main contribution of this paper. To illustrate this, on IMDb dataset, we train a Recurrent Neural Network (RNN) to achieve good privacy-vs-accuracy tradeoff, while being significantly faster than DP-SGD and with a similar memory footprint as non-private SGD. The privacy analysis of our algorithms is more involved than DP-SGD, we use the recently proposed f-DP framework of Dong et al. (2019) to prove privacy.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 21 of its 39 references that are in the map. First posted on arXiv 5 Feb 2021.

Builds on

21 of 39 references.

References · 21

  1. 2016Deep Learning with Differential Privacycited by 2,682
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  4. 2015Automatic differentiation in machine learning: a surveycited by 846
  5. 2017Renyi Differential Privacycited by 643
  6. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  7. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  8. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  9. 2016Concentrated Differential Privacycited by 139
  10. 2019Deep Learning with Gaussian Differential Privacycited by 112
  11. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 104
  12. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
9 more
  1. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  2. 2019Generative Models for Effective ML on Private, Decentralized Datasetscited by 85
  3. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  4. 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 46
  5. 2020Tight Differential Privacy for Discrete-Valued Mechanisms and for the Subsampled Gaussian Mechanism Using FFTcited by 43
  6. 2015Efficient Per-Example Gradient Computationscited by 42
  7. 2020Privacy Amplification via Random Check-Inscited by 39
  8. 2019Efficient Per-Example Gradient Computations in Convolutional Neural Networkscited by 17
  9. 2019Diversifying Reply Suggestions using a Matching-Conditional Variational Autoencodercited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 6; 2024: 3; 2025: 5; 2026: 2.

29 papers.

Sort:

  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  2. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  3. 2021Numerical Composition of Differential Privacycited by 174
  4. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  5. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  6. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  7. 2022Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributionscited by 50
  8. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  9. 2021On the Convergence and Calibration of Deep Learning with Differential Privacycited by 36
  10. 2023Selective Pre-training for Private Fine-tuningcited by 22
  11. 2022Faster Privacy Accounting via Evolving Discretizationcited by 14
  12. 2023Differentially Private SGD Without Clipping Bias: An Error-Feedback Approachcited by 13
17 more
  1. 2023On Differential Privacy and Adaptive Data Analysis with Bounded Spacecited by 9
  2. 2023Zero redundancy distributed learning with differential privacycited by 7
  3. 2021Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliabilitycited by 6
  4. 2022Private Ad Modeling with DP-SGDcited by 6
  5. 2023Private Federated Learning with Autotuned Compressioncited by 5
  6. 2023Generative Model-Based Attack on Learnable Image Encryption for Privacy-Preserving Deep Learningcited by 4
  7. 2024Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clippingcited by 2
  8. 2024Differentially Private Prototypes for Imbalanced Transfer Learningcited by 2
  9. 2025Memory-Efficient Differentially Private Training with Gradient Random Projectioncited by 2
  10. 2022Rieoptax: Riemannian Optimization in JAXcited by 1
  11. 2024Noise is All You Need: Private Second-Order Convergence of Noisy SGDcited by 1
  12. 2025Federated Learning with Differential Privacy: An Utility-Enhanced Approachcited by 1
  13. 2025PLRV-O: Advancing Differentially Private Deep Learning via Privacy Loss Random Variable Optimizationcited by 1
  14. 2026Efficient DP-SGD for LLMs with Randomized Clippingcited by 1
  15. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  16. 2025Stabilization of Perturbed Loss Function: Differential Privacy without Gradient Noisecited by 0
  17. 2026Lap2: Revisiting Laplace DP-SGD for High Dimensions via Majorization Theorycited by 0