mlsanityData to 7 Oct 2026

(Amplified) Banded Matrix Factorization: A unified approach to private training

arXiv 2306.08153 ↗First posted 13 Jun 2023cs.LG cs.CRcited by 49

Not on the map. Why?

Abstract (arXiv)

Matrix factorization (MF) mechanisms for differential privacy (DP) have substantially improved the state-of-the-art in privacy-utility-computation tradeoffs for ML applications in a variety of scenarios, but in both the centralized and federated settings there remain instances where either MF cannot be easily applied, or other algorithms provide better tradeoffs (typically, as ε becomes small). In this work, we show how MF can subsume prior state-of-the-art algorithms in both federated and centralized training settings, across all privacy budgets. The key technique throughout is the construction of MF mechanisms with banded matrices (lower-triangular matrices with at most hatb nonzero bands including the main diagonal). For cross-device federated learning (FL), this enables multiple-participations with a relaxed device participation schema compatible with practical FL infrastructure (as demonstrated by a production deployment). In the centralized setting, we prove that banded matrices enjoy the same privacy amplification results as the ubiquitous DP-SGD algorithm, but can provide strictly better performance in most scenarios -- this lets us always at least match DP-SGD, and often outperform it.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 34 of its 67 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

34 of 67 references.

References · 34

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2020Adaptive Federated Optimizationcited by 1,036
  6. 2019Towards Federated Learning at Scale: System Designcited by 826
  7. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  8. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
  9. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  10. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  11. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 183
  12. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
22 more
  1. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  2. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 152
  3. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  4. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 89
  5. 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 84
  6. 2021Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descentcited by 71
  7. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
  8. 2022Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streamscited by 59
  9. 2022Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learningcited by 48
  10. 2022Private Convex Optimization via Exponential Mechanismcited by 47
  11. 2017The Price of Differential Privacy For Online Learningcited by 44
  12. 2022Composition of Differential Privacy & Privacy Amplification by Subsamplingcited by 42
  13. 2020Privacy Amplification via Random Check-Inscited by 39
  14. 2017Learning Differentially Private Recurrent Language Modelscited by 32
  15. 2019The Power of Factorization Mechanisms in Local and Central Differential Privacycited by 30
  16. 2018Optimizing error of high-dimensional statistical queries under differential privacycited by 29
  17. 2020Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descentcited by 27
  18. 2021Adapting to Function Difficulty and Growth Conditions in Private Optimizationcited by 19
  19. 2022Private Online Prediction from Experts: Separations and Faster Ratescited by 17
  20. 2021HDMM: Optimizing error of high-dimensional statistical queries under differential privacycited by 11
  21. 2023Near-Optimal Algorithms for Private Online Optimization in the Realizable Regimecited by 9
  22. 2016Convex Optimization for Linear Query Processing under Approximate Differential Privacycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 19; 2025: 15; 2026: 11.

49 papers.

Sort:

  1. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 29
  2. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 28
  3. 2023Correlated Noise Provably Beats Independent Noise for Differentially Private Learningcited by 23
  4. 2023Privacy Amplification for Matrix Mechanismscited by 17
  5. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  6. 2024Efficient and Near-Optimal Noise Generation for Streaming Differential Privacycited by 17
  7. 2024Federated Learning in Practice: Reflections and Projectionscited by 15
  8. 2023A Smooth Binary Mechanism for Efficient Private Continual Observationcited by 12
  9. 2024Banded Square Root Matrix Factorization for Differentially Private Model Trainingcited by 12
  10. 2024A Hassle-free Algorithm for Private Learning in Practice: Don't Use Tree Aggregation, Use BLTscited by 9
  11. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  12. 2025Back to Square Roots: An Optimal Bound on the Matrix Factorization Error for Multi-Epoch Differentially Private SGDcited by 7
37 more
  1. 2024To Shuffle or not to Shuffle: Auditing DP-SGD with Shufflingcited by 6
  2. 2024Locally Differentially Private Online Federated Learning With Correlated Noisecited by 6
  3. 2025On Design Principles for Private Adaptive Optimizerscited by 6
  4. 2025Continual Release Moment Estimation with Differential Privacycited by 5
  5. 2024Confidential Federated Computationscited by 4
  6. 2024Correlated Privacy Mechanisms for Differentially Private Distributed Mean Estimationcited by 4
  7. 2024Momentum Approximation in Asynchronous Private Federated Learningcited by 3
  8. 2024DMM: Distributed Matrix Mechanism for Differentially-Private Federated Learning Based on Constant-Overhead Linear Secret Resharingcited by 3
  9. 2025Empirical Privacy Variancecited by 3
  10. 2025Binned Group Algebra Factorization for Differentially Private Continual Countingcited by 3
  11. 2025An Inversion Theorem for Buffered Linear Toeplitz (BLT) Matrices and Applications to Streaming Differential Privacycited by 3
  12. 2026Sampling-Free Privacy Accounting for Matrix Mechanisms under Random Allocationcited by 3
  13. 2024Tight Group-Level DP Guarantees for DP-SGD with Sampling via Mixture of Gaussians Mechanismscited by 2
  14. 2024Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clippingcited by 2
  15. 2025An Optimization Framework for Differentially Private Sparse Fine-Tuningcited by 2
  16. 2025Synthesizing and Adapting Error Correction Data for Mobile Large Language Model Applicationscited by 2
  17. 2025Private Continual Counting of Unbounded Streamscited by 2
  18. 2025Unified Privacy Guarantees for Decentralized Learning via Matrix Factorizationcited by 2
  19. 2025Learning Rate Scheduling with Matrix Factorization for Private Trainingcited by 2
  20. 2026JAX-Privacy: A library for differentially private machine learningcited by 2
  21. 2026High-Probability Bounds for SGD under the Polyak-Lojasiewicz Condition with Markovian Noisecited by 2
  22. 2023Robust and Actively Secure Serverless Collaborative Learningcited by 1
  23. 2024Efficient Language Model Architectures for Differentially Private Federated Learningcited by 1
  24. 2024Debiasing Federated Learning with Correlated Client Participationcited by 1
  25. 2024Towards Privacy-Preserving Medical Imaging: Federated Learning with Differential Privacy and Secure Aggregation Using a Modified ResNet Architecturecited by 1
  26. 2025Correlating Cross-Iteration Noise for DP-SGD using Model Curvaturecited by 1
  27. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  28. 2026DP-λCGD: Efficient Noise Correlation for Differentially Private Model Trainingcited by 1
  29. 2026FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correctioncited by 1
  30. 2025It's My Data Too: Private ML for Datasets with Multi-User Training Examplescited by 0
  31. 2025Prediction with Expert Advice under Local Differential Privacycited by 0
  32. 2026Matrix Factorization for Practical Continual Mean Estimation Under User-Level Differential Privacycited by 0
  33. 2026Secure and Privacy-Preserving Vertical Federated Learningcited by 0
  34. 2026Beyond Square Roots: A Memory-Efficient Explicit Factorization for Multi-Epoch Private Learningcited by 0
  35. 2026When Do Fewer Coordinates Suffice in DP-SGD?cited by 0
  36. 2026Revisiting Privacy Amplification by Subsampling in Selective Release DPSGDcited by 0
  37. 2026High-Probability PL-SGD with Markovian Noise: Optimal Mixing and Tail Dependencecited by 0