mlsanityData to 14 Oct 2026

Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning

arXiv 2211.06530 ↗First posted 12 Nov 2022cs.LG cs.CR cs.DScited by 48

Not on the map. Why?

Abstract (arXiv)

We introduce new differentially private (DP) mechanisms for gradient-based machine learning (ML) with multiple passes (epochs) over a dataset, substantially improving the achievable privacy-utility-computation tradeoffs. We formalize the problem of DP mechanisms for adaptive streams with multiple participations and introduce a non-trivial extension of online matrix factorization DP mechanisms to our setting. This includes establishing the necessary theory for sensitivity calculations and efficient computation of optimal matrices. For some applications like >!! 10,000 SGD steps, applying these optimal techniques becomes computationally expensive. We thus design an efficient Fourier-transform-based mechanism with only a minor utility loss. Extensive empirical evaluation on both example-level DP for image classification and user-level DP for language modeling demonstrate substantial improvements over all previous methods, including the widely-used DP-SGD . Though our primary application is to ML, our main DP results are applicable to arbitrary linear queries and hence may have much broader applicability.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 16 of its 50 references that are in the map. First posted on arXiv 12 Nov 2022.

Builds on

16 of 50 references.

References · 16

  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2020Adaptive Federated Optimizationcited by 1,046
  4. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  5. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 346
  6. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 236
  7. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  8. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  9. 2018Amplification by Shuffling: From Local to Central Differential Privacy via Anonymitycited by 164
  10. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 107
  11. 2020Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shufflingcited by 96
  12. 2022Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streamscited by 59
4 more
  1. 2019The Power of Factorization Mechanisms in Local and Central Differential Privacycited by 30
  2. 2018Optimizing error of high-dimensional statistical queries under differential privacycited by 29
  3. 2022Almost Tight Error Bounds on Differentially Private Continual Countingcited by 26
  4. 2013Nearly Optimal Private Convolutioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 12; 2025: 14; 2026: 6.

48 papers.

Sort:

  1. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  2. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 66
  3. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  4. 2023Privacy Side Channels in Machine Learning Systemscited by 40
  5. 2023One-shot Empirical Privacy Estimation for Federated Learningcited by 36
  6. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 32
  7. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  8. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 29
  9. 2023Correlated Noise Provably Beats Independent Noise for Differentially Private Learningcited by 23
  10. 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
  11. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  12. 2023Privacy Amplification for Matrix Mechanismscited by 17
36 more
  1. 2023A Unifying Framework for Differentially Private Sums under Continual Observationcited by 15
  2. 2023A Smooth Binary Mechanism for Efficient Private Continual Observationcited by 12
  3. 2024Banded Square Root Matrix Factorization for Differentially Private Model Trainingcited by 12
  4. 2024A Hassle-free Algorithm for Private Learning in Practice: Don't Use Tree Aggregation, Use BLTscited by 9
  5. 2024Streaming Private Continual Counting via Binningcited by 7
  6. 2025Back to Square Roots: An Optimal Bound on the Matrix Factorization Error for Multi-Epoch Differentially Private SGDcited by 7
  7. 2025On Design Principles for Private Adaptive Optimizerscited by 6
  8. 2023Private Federated Learning with Autotuned Compressioncited by 5
  9. 2025Continual Release Moment Estimation with Differential Privacycited by 5
  10. 2024Improved Communication-Privacy Trade-offs in L2 Mean Estimation under Streaming Differential Privacycited by 4
  11. 2023General Gaussian Noise Mechanisms and Their Optimality for Unbiased Mean Estimationcited by 3
  12. 2024DMM: Distributed Matrix Mechanism for Differentially-Private Federated Learning Based on Constant-Overhead Linear Secret Resharingcited by 3
  13. 2025Empirical Privacy Variancecited by 3
  14. 2025Binned Group Algebra Factorization for Differentially Private Continual Countingcited by 3
  15. 2025Normalized Square Root: Sharper Matrix Factorization Bounds for Differentially Private Continual Countingcited by 3
  16. 2026Sampling-Free Privacy Accounting for Matrix Mechanisms under Random Allocationcited by 3
  17. 2026DP-λCGD: Efficient Noise Correlation for Differentially Private Model Trainingcited by 3
  18. 2023An Empirical Evaluation of Federated Contextual Bandit Algorithmscited by 2
  19. 2024Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clippingcited by 2
  20. 2025Private Continual Counting of Unbounded Streamscited by 2
  21. 2025Unified Privacy Guarantees for Decentralized Learning via Matrix Factorizationcited by 2
  22. 2025Learning Rate Scheduling with Matrix Factorization for Private Trainingcited by 2
  23. 2025Operator-Theoretic Framework for Gradient-Free Federated Learningcited by 2
  24. 2023Robust and Actively Secure Serverless Collaborative Learningcited by 1
  25. 2024Optimal Rates for O(1)-Smooth DP-SCO with a Single Epoch and Large Batchescited by 1
  26. 2024Differentially Private Multimodal Laplacian Dropout (DP-MLD) for EEG Representative Learningcited by 1
  27. 2024Towards Privacy-Preserving Medical Imaging: Federated Learning with Differential Privacy and Secure Aggregation Using a Modified ResNet Architecturecited by 1
  28. 2024Data value estimation on private gradientscited by 1
  29. 2025Securing Private Federated Learning in a Malicious Setting: A Scalable TEE-Based Approach with Client Auditingcited by 1
  30. 2025Correlating Cross-Iteration Noise for DP-SGD using Model Curvaturecited by 1
  31. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  32. 2026Efficient DP-SGD for LLMs with Randomized Clippingcited by 1
  33. 2025It's My Data Too: Private ML for Datasets with Multi-User Training Examplescited by 0
  34. 2026Matrix Factorization for Practical Continual Mean Estimation Under User-Level Differential Privacycited by 0
  35. 2026Secure and Privacy-Preserving Vertical Federated Learningcited by 0
  36. 2026Beyond Square Roots: A Memory-Efficient Explicit Factorization for Multi-Epoch Private Learningcited by 0