mlsanityData to 7 Oct 2026

Multi-Message Shuffled Privacy in Federated Learning

arXiv 2302.11152 ↗First posted 22 Feb 2023cs.LG cs.CRcited by 2

Not on the map. Why?

Abstract (arXiv)

We study differentially private distributed optimization under communication constraints. A server using SGD for optimization aggregates the client-side local gradients for model updates using distributed mean estimation (DME). We develop a communication-efficient private DME, using the recently developed multi-message shuffled (MMS) privacy framework. We analyze our proposed DME scheme to show that it achieves the order-optimal privacy-communication-performance tradeoff resolving an open question in [1], whether the shuffled models can improve the tradeoff obtained in Secure Aggregation. This also resolves an open question on the optimal trade-off for private vector sum in the MMS model. We achieve it through a novel privacy mechanism that non-uniformly allocates privacy at different resolutions of the local gradient vectors. These results are directly applied to give guarantees on private distributed learning algorithms using this for private gradient aggregation iteratively. We also numerically evaluate the private DME algorithms.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 41 of its 65 references that are in the map. First posted on arXiv 22 Feb 2023.

Builds on

41 of 65 references.

References · 41

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2019Deep Leakage from Gradientscited by 1,004
  7. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  8. 2017Renyi Differential Privacycited by 643
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  10. 2014RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Responsecited by 262
  11. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  12. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
29 more
  1. 2018Amplification by Shuffling: From Local to Central Differential Privacy via Anonymitycited by 163
  2. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
  3. 2018cpSGD: Communication-efficient and differentially-private distributed SGDcited by 157
  4. 2017Collecting Telemetry Data Privatelycited by 147
  5. 2016Distributed Mean Estimation with Limited Communicationcited by 134
  6. 2020The Discrete Gaussian for Differential Privacycited by 134
  7. 2018Protection Against Reconstruction and Its Applications in Private Federated Learningcited by 127
  8. 2021The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregationcited by 127
  9. 2018Distributed Differential Privacy via Shufflingcited by 115
  10. 2020Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shufflingcited by 96
  11. 2019Hypothesis Testing Interpretations and Renyi Differential Privacycited by 84
  12. 2013Fingerprinting Codes and the Price of Approximate Differential Privacycited by 81
  13. 2021Learning with User-Level Privacycited by 72
  14. 2020Breaking the Communication-Privacy-Accuracy Trilemmacited by 68
  15. 2016Discrete Distribution Estimation under Local Privacycited by 58
  16. 2020Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluationcited by 39
  17. 2022The Fundamental Price of Secure Aggregation in Differentially Private Federated Learningcited by 39
  18. 2022Optimal Algorithms for Mean Estimation under Local Differential Privacycited by 36
  19. 2020Private Summation in the Multi-Message Shuffle Modelcited by 34
  20. 2022Stronger Privacy Amplification by Shuffling for R'enyi and Approximate Differential Privacycited by 29
  21. 2021Lossless Compression of Efficient Private Local Randomizerscited by 27
  22. 2021Locally Private k-Means in One Roundcited by 19
  23. 2021Differentially Private Aggregation in the Shuffle Model: Almost Central Accuracy in Almost a Single Messagecited by 18
  24. 2021Shuffle Private Stochastic Convex Optimizationcited by 17
  25. 2022Privacy-Aware Compression for Federated Data Analysiscited by 17
  26. 2021On the Renyi Differential Privacy of the Shuffle Modelcited by 16
  27. 2021Private Counting from Anonymous Messages: Near-Optimal Accuracy with Vanishing Communication Overheadcited by 16
  28. 2020Pure Differentially Private Summation from Anonymous Messagescited by 15
  29. 2021Renyi Differential Privacy of the Subsampled Shuffle Model in Distributed Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Privacy Amplification via Compression: Achieving the Optimal Privacy-Accuracy-Communication Trade-off in Distributed Mean Estimationcited by 22
  2. 2024Improved Communication-Privacy Trade-offs in L2 Mean Estimation under Streaming Differential Privacycited by 4