mlsanityData to 14 Oct 2026

Practical and Private (Deep) Learning without Sampling or Shuffling

arXiv 2103.00039 ↗First posted 26 Feb 2021cs.CR cs.LGcited by 170

Not on the map. Why?

Abstract (arXiv)

We consider training models with differential privacy (DP) using mini-batch gradients. The existing state-of-the-art, Differentially Private Stochastic Gradient Descent (DP-SGD), requires privacy amplification by sampling or shuffling to obtain the best privacy/accuracy/computation trade-offs. Unfortunately, the precise requirements on exact sampling and shuffling can be hard to obtain in important practical scenarios, particularly federated learning (FL). We design and analyze a DP variant of Follow-The-Regularized-Leader (DP-FTRL) that compares favorably (both theoretically and empirically) to amplified DP-SGD, while allowing for much more flexible data access patterns. DP-FTRL does not use any form of privacy amplification. The code is available at https://github.com/google-research/federated/tree/master/dp_ftrl and https://github.com/google-research/DP-FTRL .

Key facts. Cited by 170 papers in the map (data through 14 Oct 2026). Builds on 29 of its 77 references that are in the map. First posted on arXiv 26 Feb 2021.

Builds on

29 of 77 references.

References · 29

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2016Deep Learning with Differential Privacycited by 2,728
  4. 2020Adaptive Federated Optimizationcited by 1,046
  5. 2019Towards Federated Learning at Scale: System Designcited by 832
  6. 2019Introduction to Online Convex Optimizationcited by 803
  7. 2017Renyi Differential Privacycited by 650
  8. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 236
  9. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  10. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  11. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 218
  12. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 199
17 more
  1. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 186
  2. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 173
  3. 2018Amplification by Shuffling: From Local to Central Differential Privacy via Anonymitycited by 164
  4. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 153
  5. 2020The Discrete Gaussian for Differential Privacycited by 135
  6. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 122
  7. 2018Privacy Amplification by Iterationcited by 100
  8. 2020Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shufflingcited by 96
  9. 2018A General Approach to Adding Differential Privacy to Iterative Training Procedurescited by 84
  10. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  11. 2020Training Production Language Models without Memorizing User Datacited by 68
  12. 2017The Price of Differential Privacy For Online Learningcited by 45
  13. 2020Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluationcited by 39
  14. 2020Privacy Amplification via Random Check-Inscited by 39
  15. 2020Understanding Unintended Memorization in Federated Learningcited by 36
  16. 2020A Better Bound Gives a Hundred Rounds: Enhanced Privacy Guarantees via f-Divergencescited by 21
  17. 2017Online Learning via the Differential Privacy Lenscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 26; 2023: 33; 2024: 49; 2025: 31; 2026: 17.

170 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 315
  2. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  3. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  4. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  5. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  6. 2021The Skellam Mechanism for Differentially Private Federated Learningcited by 68
  7. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 66
  8. 2022Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streamscited by 59
  9. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  10. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  11. 2024Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearningcited by 49
  12. 2022Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)cited by 48
38 more
  1. 2022Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learningcited by 48
  2. 2021Node-Level Differentially Private Graph Neural Networkscited by 43
  3. 2022Private Adaptive Optimization with Side Informationcited by 38
  4. 2021On the Convergence and Calibration of Deep Learning with Differential Privacycited by 36
  5. 2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
  6. 2023One-shot Empirical Privacy Estimation for Federated Learningcited by 36
  7. 2023User Inference Attacks on Large Language Modelscited by 36
  8. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  9. 2021DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?cited by 33
  10. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 32
  11. 2021Quantum machine learning with differential privacycited by 31
  12. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  13. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  14. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  15. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 29
  16. 2022Almost Tight Error Bounds on Differentially Private Continual Countingcited by 26
  17. 2022CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learningcited by 24
  18. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  19. 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 24
  20. 2022Constant matters: Fine-grained Complexity of Differentially Private Continual Observationcited by 23
  21. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  22. 2023Correlated Noise Provably Beats Independent Noise for Differentially Private Learningcited by 23
  23. 2024How Private are DP-SGD Implementations?cited by 22
  24. 2023Have it your way: Individualized Privacy Assignment for DP-SGDcited by 21
  25. 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
  26. 2023Practical Differentially Private Hyperparameter Tuning with Subsamplingcited by 19
  27. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  28. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  29. 2024The Privacy Power of Correlated Noise in Decentralized Learningcited by 18
  30. 2021Shuffle Private Stochastic Convex Optimizationcited by 17
  31. 2022Private Online Prediction from Experts: Separations and Faster Ratescited by 17
  32. 2023Privacy Amplification for Matrix Mechanismscited by 17
  33. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  34. 2024Efficient and Near-Optimal Noise Generation for Streaming Differential Privacycited by 17
  35. 2022Near-Optimal Differentially Private Reinforcement Learningcited by 16
  36. 2023Reconstructing Individual Data Points in Federated Learning Hardened with Differential Privacy and Secure Aggregationcited by 16
  37. 2024Scalable DP-SGD: Shuffling vs. Poisson Subsamplingcited by 16
  38. 2025VaultGemma: A Differentially Private Gemma Modelcited by 16

50 most cited of 170 All 170 (JSON)