# Gradient Disaggregation: Breaking Privacy in Federated Learning by Reconstructing the User Participant Matrix

arXiv: https://arxiv.org/abs/2106.06089 · HTML: https://mlsanity.com/paper/2106.06089/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning](https://mlsanity.com/paper/1702.07464/) (2017)
- [Attack of the Tails: Yes, You Really Can Backdoor Federated Learning](https://mlsanity.com/paper/2007.05084/) (2020)
- [Federated Optimization:Distributed Optimization Beyond the Datacenter](https://mlsanity.com/paper/1511.03575/) (2015)
- [Threats to Federated Learning: A Survey](https://mlsanity.com/paper/2003.02133/) (2020)
- [Mitigating Sybils in Federated Learning Poisoning](https://mlsanity.com/paper/1808.04866/) (2018)
- [Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning](https://mlsanity.com/paper/2002.04156/) (2020)
- [A Framework for Evaluating Gradient Leakage Attacks in Federated Learning](https://mlsanity.com/paper/2004.10397/) (2020)
- [Matrix factorization with Binary Components](https://mlsanity.com/paper/1401.6024/) (2014)

## Built on by (most cited first, 30 of 43)

- [Federated Learning with Buffered Asynchronous Aggregation](https://mlsanity.com/paper/2106.06639/) (2021)
- [Federated Learning Enables Big Data for Rare Cancer Boundary Detection](https://mlsanity.com/paper/2204.10836/) (2022)
- [Fishing for User Data in Large-Batch Federated Learning via Gradient Magnification](https://mlsanity.com/paper/2202.00580/) (2022)
- [Eluding Secure Aggregation in Federated Learning via Model Inconsistency](https://mlsanity.com/paper/2111.07380/) (2021)
- [Recent Advances on Federated Learning: A Systematic Survey](https://mlsanity.com/paper/2301.01299/) (2023)
- [A Survey on Gradient Inversion: Attacks, Defenses and Future Directions](https://mlsanity.com/paper/2206.07284/) (2022)
- [Gradient Obfuscation Gives a False Sense of Security in Federated Learning](https://mlsanity.com/paper/2206.04055/) (2022)
- [Personalized Federated Learning under Mixture of Distributions](https://mlsanity.com/paper/2305.01068/) (2023)
- [LOKI: Large-scale Data Reconstruction Attack against Federated Learning through Model Manipulation](https://mlsanity.com/paper/2303.12233/) (2023)
- [SNAP: Efficient Extraction of Private Properties with Poisoning](https://mlsanity.com/paper/2208.12348/) (2022)
- [FLASHE: Additively Symmetric Homomorphic Encryption for Cross-Silo Federated Learning](https://mlsanity.com/paper/2109.00675/) (2021)
- [Trustworthy Distributed AI Systems: Robustness, Privacy, and Governance](https://mlsanity.com/paper/2402.01096/) (2024)
- [The Federation Strikes Back: A Survey of Federated Learning Privacy Attacks, Defenses, Applications, and Policy Landscape](https://mlsanity.com/paper/2405.03636/) (2024)
- [Privacy-preserving Decentralized Federated Learning over Time-varying Communication Graph](https://mlsanity.com/paper/2210.00325/) (2022)
- [Surrogate Model Extension (SME): A Fast and Accurate Weight Update Attack on Federated Learning](https://mlsanity.com/paper/2306.00127/) (2023)
- [Client-specific Property Inference against Secure Aggregation in Federated Learning](https://mlsanity.com/paper/2303.03908/) (2023)
- [The Resource Problem of Using Linear Layer Leakage Attack in Federated Learning](https://mlsanity.com/paper/2303.14868/) (2023)
- [Hiding in Plain Sight: Disguising Data Stealing Attacks in Federated Learning](https://mlsanity.com/paper/2306.03013/) (2023)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimation](https://mlsanity.com/paper/2406.12815/) (2024)
- [ScionFL: Efficient and Robust Secure Quantized Aggregation](https://mlsanity.com/paper/2210.07376/) (2022)
- [Theoretically Principled Federated Learning for Balancing Privacy and Utility](https://mlsanity.com/paper/2305.15148/) (2023)
- [Secure Aggregation is Not Private Against Membership Inference Attacks](https://mlsanity.com/paper/2403.17775/) (2024)
- [Breaking Secure Aggregation: Label Leakage from Aggregated Gradients in Federated Learning](https://mlsanity.com/paper/2406.15731/) (2024)
- [Maximum Knowledge Orthogonality Reconstruction with Gradients in Federated Learning](https://mlsanity.com/paper/2310.19222/) (2023)
- [Analyzing Inference Privacy Risks Through Gradients in Machine Learning](https://mlsanity.com/paper/2408.16913/) (2024)
- [Perfectly Accurate Membership Inference by a Dishonest Central Server in Federated Learning](https://mlsanity.com/paper/2203.16463/) (2022)
- [SRATTA : Sample Re-ATTribution Attack of Secure Aggregation in Federated Learning](https://mlsanity.com/paper/2306.07644/) (2023)
- [Linkage on Security, Privacy and Fairness in Federated Learning: New Balances and New Perspectives](https://mlsanity.com/paper/2406.10884/) (2024)
- [Kick Bad Guys Out! Conditionally Activated Anomaly Detection in Federated Learning with Zero-Knowledge Proof Verification](https://mlsanity.com/paper/2310.04055/) (2023)

## Abstract (arXiv)

We show that aggregated model updates in federated learning may be insecure. An untrusted central server may disaggregate user updates from sums of updates across participants given repeated observations, enabling the server to recover privileged information about individual users' private training data via traditional gradient inference attacks. Our method revolves around reconstructing participant information (e.g: which rounds of training users participated in) from aggregated model updates by leveraging summary information from device analytics commonly used to monitor, debug, and manage federated learning systems. Our attack is parallelizable and we successfully disaggregate user updates on settings with up to thousands of participants. We quantitatively and qualitatively demonstrate significant improvements in the capability of various inference attacks on the disaggregated updates. Our attack enables the attribution of learned properties to individual users, violating anonymity, and shows that a determined central server may undermine the secure aggregation protocol to break individual users' data privacy in federated learning.