# Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning

arXiv: https://arxiv.org/abs/1702.07464 · HTML: https://mlsanity.com/paper/1702.07464/

## Key facts

- Cited by 334 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2017.

## Built on by (most cited first, 30 of 334)

- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [See through Gradients: Image Batch Recovery via GradInversion](https://mlsanity.com/paper/2104.07586/) (2021)
- [Zero-Shot Machine Unlearning](https://mlsanity.com/paper/2201.05629/) (2022)
- [Reconstructing Training Data from Trained Neural Networks](https://mlsanity.com/paper/2206.07758/) (2022)
- [When the Curious Abandon Honesty: Federated Learning Is Not Private](https://mlsanity.com/paper/2112.02918/) (2021)
- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [Gradient Inversion with Generative Image Prior](https://mlsanity.com/paper/2110.14962/) (2021)
- [Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets](https://mlsanity.com/paper/2204.00032/) (2022)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [PPFL: Privacy-preserving Federated Learning with Trusted Execution Environments](https://mlsanity.com/paper/2104.14380/) (2021)
- [MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation](https://mlsanity.com/paper/2110.01406/) (2021)
- [Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning](https://mlsanity.com/paper/2102.12677/) (2021)
- [Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges](https://mlsanity.com/paper/2201.08135/) (2022)
- [CAFE: Catastrophic Data Leakage in Vertical Federated Learning](https://mlsanity.com/paper/2110.15122/) (2021)
- [Variational Model Inversion Attacks](https://mlsanity.com/paper/2201.10787/) (2022)
- [Auditing Privacy Defenses in Federated Learning via Generative Gradient Leakage](https://mlsanity.com/paper/2203.15696/) (2022)
- [Eluding Secure Aggregation in Federated Learning via Model Inconsistency](https://mlsanity.com/paper/2111.07380/) (2021)
- [From Distributed Machine Learning to Federated Learning: A Survey](https://mlsanity.com/paper/2104.14362/) (2021)
- [Provably Secure Federated Learning against Malicious Clients](https://mlsanity.com/paper/2102.01854/) (2021)
- [On Memorization in Probabilistic Deep Generative Models](https://mlsanity.com/paper/2106.03216/) (2021)
- [PUMA: Performance Unchanged Model Augmentation for Training Data Removal](https://mlsanity.com/paper/2203.00846/) (2022)
- [Socially Responsible AI Algorithms: Issues, Purposes, and Challenges](https://mlsanity.com/paper/2101.02032/) (2021)
- [FedLab: A Flexible Federated Learning Framework](https://mlsanity.com/paper/2107.11621/) (2021)
- [FederatedScope: A Flexible Federated Learning Platform for Heterogeneity](https://mlsanity.com/paper/2204.05011/) (2022)
- [Federated Continual Learning via Knowledge Fusion: A Survey](https://mlsanity.com/paper/2312.16475/) (2023)
- [Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping](https://mlsanity.com/paper/2212.01539/) (2022)
- [GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization](https://mlsanity.com/paper/2308.04699/) (2023)
- [Survey: Leakage and Privacy at Inference Time](https://mlsanity.com/paper/2107.01614/) (2021)
- [Do Gradient Inversion Attacks Make Federated Learning Unsafe?](https://mlsanity.com/paper/2202.06924/) (2022)
- [Gradient Disaggregation: Breaking Privacy in Federated Learning by Reconstructing the User Participant Matrix](https://mlsanity.com/paper/2106.06089/) (2021)

## Abstract (arXiv)

Deep Learning has recently become hugely popular in machine learning, providing significant improvements in classification accuracy in the presence of highly-structured and large databases. Researchers have also considered privacy implications of deep learning. Models are typically trained in a centralized manner with all the data being processed by the same training algorithm. If the data is a collection of users' private data, including habits, personal pictures, geographical positions, interests, and more, the centralized server will have access to sensitive information that could potentially be mishandled. To tackle this problem, collaborative deep learning models have recently been proposed where parties locally train their deep learning structures and only share a subset of the parameters in the attempt to keep their respective training sets private. Parameters can also be obfuscated via differential privacy (DP) to make information extraction even more challenging, as proposed by Shokri and Shmatikov at CCS'15. Unfortunately, we show that any privacy-preserving collaborative deep learning is susceptible to a powerful attack that we devise in this paper. In particular, we show that a distributed, federated, or decentralized deep learning approach is fundamentally broken and does not protect the training sets of honest participants. The attack we developed exploits the real-time nature of the learning process that allows the adversary to train a Generative Adversarial Network (GAN) that generates prototypical samples of the targeted training set that was meant to be private (the samples generated by the GAN are intended to come from the same distribution as the training data). Interestingly, we show that record-level DP applied to the shared parameters of the model, as suggested in previous work, is ineffective (i.e., record-level DP is not designed to address our attack).