mlsanityData to 7 Oct 2026

Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streams

arXiv 2202.08312 ↗First posted 16 Feb 2022cs.LG math.OCcited by 59

Not on the map. Why?

Abstract (arXiv)

Motivated by recent applications requiring differential privacy over adaptive streams, we investigate the question of optimal instantiations of the matrix mechanism in this setting. We prove fundamental theoretical results on the applicability of matrix factorizations to adaptive streams, and provide a parameter-free fixed-point algorithm for computing optimal factorizations. We instantiate this framework with respect to concrete matrices which arise naturally in machine learning, and train user-level differentially private models with the resulting optimal mechanisms, yielding significant improvements in a notable problem in federated learning with user-level differential privacy.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 25 of its 81 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

25 of 81 references.

References · 25

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2021Convex optimizationcited by 2,479
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2020Adaptive Federated Optimizationcited by 1,036
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  7. 2017Renyi Differential Privacycited by 643
  8. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  9. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
  10. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  11. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  12. 2016Concentrated Differential Privacycited by 139
13 more
  1. 2021Federated Reconstruction: Partially Local Federated Learningcited by 78
  2. 2021On Large-Cohort Training for Federated Learningcited by 73
  3. 2020Training Production Language Models without Memorizing User Datacited by 68
  4. 2017The Price of Differential Privacy For Online Learningcited by 44
  5. 2020Tight Differential Privacy for Discrete-Valued Mechanisms and for the Subsampled Gaussian Mechanism Using FFTcited by 43
  6. 2021The Price of Differential Privacy under Continual Observationcited by 32
  7. 2019The Power of Factorization Mechanisms in Local and Central Differential Privacycited by 30
  8. 2018Optimizing error of high-dimensional statistical queries under differential privacycited by 29
  9. 2019That which we call privatecited by 14
  10. 2021Differentially Private Histograms under Continual Observation: Streaming Selection into the Unknowncited by 14
  11. 2018Answering Range Queries Under Local Differential Privacycited by 12
  12. 2021HDMM: Optimizing error of high-dimensional statistical queries under differential privacycited by 11
  13. 2018Local Differential Privacy for Evolving Datacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 13; 2024: 16; 2025: 18; 2026: 7.

59 papers.

Sort:

  1. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  2. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  3. 2022Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learningcited by 48
  4. 2023One-shot Empirical Privacy Estimation for Federated Learningcited by 35
  5. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  6. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 29
  7. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 28
  8. 2022Almost Tight Error Bounds on Differentially Private Continual Countingcited by 25
  9. 2022Constant matters: Fine-grained Complexity of Differentially Private Continual Observationcited by 23
  10. 2023Correlated Noise Provably Beats Independent Noise for Differentially Private Learningcited by 23
  11. 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
  12. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
38 more
  1. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  2. 2023Privacy Amplification for Matrix Mechanismscited by 17
  3. 2023A Unifying Framework for Differentially Private Sums under Continual Observationcited by 15
  4. 2023A Smooth Binary Mechanism for Efficient Private Continual Observationcited by 12
  5. 2024Banded Square Root Matrix Factorization for Differentially Private Model Trainingcited by 12
  6. 2024A Hassle-free Algorithm for Private Learning in Practice: Don't Use Tree Aggregation, Use BLTscited by 9
  7. 2024DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reductioncited by 8
  8. 2024Streaming Private Continual Counting via Binningcited by 7
  9. 2025Back to Square Roots: An Optimal Bound on the Matrix Factorization Error for Multi-Epoch Differentially Private SGDcited by 7
  10. 2023PLAN: Variance-Aware Private Mean Estimationcited by 6
  11. 2024Locally Differentially Private Online Federated Learning With Correlated Noisecited by 6
  12. 2025On Design Principles for Private Adaptive Optimizerscited by 6
  13. 2025Continual Release Moment Estimation with Differential Privacycited by 5
  14. 2024Improved Communication-Privacy Trade-offs in L2 Mean Estimation under Streaming Differential Privacycited by 4
  15. 2023Private Statistical Estimation of Many Quantilescited by 3
  16. 2024Momentum Approximation in Asynchronous Private Federated Learningcited by 3
  17. 2024DMM: Distributed Matrix Mechanism for Differentially-Private Federated Learning Based on Constant-Overhead Linear Secret Resharingcited by 3
  18. 2025Empirical Privacy Variancecited by 3
  19. 2025Binned Group Algebra Factorization for Differentially Private Continual Countingcited by 3
  20. 2025An Inversion Theorem for Buffered Linear Toeplitz (BLT) Matrices and Applications to Streaming Differential Privacycited by 3
  21. 2025GeoClip: Geometry-Aware Clipping for Differentially Private SGDcited by 3
  22. 2025Normalized Square Root: Sharper Matrix Factorization Bounds for Differentially Private Continual Countingcited by 3
  23. 2026Sampling-Free Privacy Accounting for Matrix Mechanisms under Random Allocationcited by 3
  24. 2023Concurrent Shuffle Differential Privacy Under Continual Observationcited by 2
  25. 2024Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clippingcited by 2
  26. 2025Differential Privacy Analysis of Decentralized Gossip Averaging under Varying Threat Modelscited by 2
  27. 2025Private Continual Counting of Unbounded Streamscited by 2
  28. 2025Unified Privacy Guarantees for Decentralized Learning via Matrix Factorizationcited by 2
  29. 2025Learning Rate Scheduling with Matrix Factorization for Private Trainingcited by 2
  30. 2022Recycling Scraps: Improving Private Learning by Leveraging Intermediate Checkpointscited by 1
  31. 2024Adaptive Data Analysis for Growing Datacited by 1
  32. 2024Optimal Rates for O(1)-Smooth DP-SCO with a Single Epoch and Large Batchescited by 1
  33. 2024Data value estimation on private gradientscited by 1
  34. 2025DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Schedulingcited by 1
  35. 2025Correlating Cross-Iteration Noise for DP-SGD using Model Curvaturecited by 1
  36. 2025Operator-Theoretic Framework for Gradient-Free Federated Learningcited by 1
  37. 2026DP-λCGD: Efficient Noise Correlation for Differentially Private Model Trainingcited by 1
  38. 2024Differentially Private Online Federated Learning with Correlated Noisecited by 0

50 most cited of 59 All 59 (JSON)