# Scale-MIA: A Scalable Model Inversion Attack against Secure Federated Learning via Latent Space Reconstruction

arXiv: https://arxiv.org/abs/2311.05808 · HTML: https://mlsanity.com/paper/2311.05808/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 66 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [See through Gradients: Image Batch Recovery via GradInversion](https://mlsanity.com/paper/2104.07586/) (2021)
- [Robust Aggregation for Federated Learning](https://mlsanity.com/paper/1912.13445/) (2019)
- [Systematic Evaluation of Privacy Risks of Machine Learning Models](https://mlsanity.com/paper/2003.10595/) (2020)
- [Grammar Variational Autoencoder](https://mlsanity.com/paper/1703.01925/) (2017)
- [Evaluating Gradient Inversion Attacks and Defenses in Federated Learning](https://mlsanity.com/paper/2112.00059/) (2021)
- [Data Synthesis based on Generative Adversarial Networks](https://mlsanity.com/paper/1806.03384/) (2018)
- [Variational Autoencoder for Deep Learning of Images, Labels and Captions](https://mlsanity.com/paper/1609.08976/) (2016)
- [Recent Advances in Autoencoder-Based Representation Learning](https://mlsanity.com/paper/1812.05069/) (2018)
- [Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models](https://mlsanity.com/paper/2110.13057/) (2021)
- [R-GAP: Recursive Gradient Attack on Privacy](https://mlsanity.com/paper/2010.07733/) (2020)
- [Fishing for User Data in Large-Batch Federated Learning via Gradient Magnification](https://mlsanity.com/paper/2202.00580/) (2022)
- [A Stochastic Quasi-Newton Method for Large-Scale Optimization](https://mlsanity.com/paper/1401.7020/) (2014)
- [Eluding Secure Aggregation in Federated Learning via Model Inconsistency](https://mlsanity.com/paper/2111.07380/) (2021)
- [FastSecAgg: Scalable Secure Aggregation for Privacy-Preserving Federated Learning](https://mlsanity.com/paper/2009.11248/) (2020)

## Built on by (most cited first, 21 of 21)

- [Exploring the Vulnerabilities of Federated Learning: A Deep Dive into Gradient Inversion Attacks](https://mlsanity.com/paper/2503.11514/) (2025)
- [State-of-the-Art Approaches to Enhancing Privacy Preservation of Machine Learning Datasets: A Survey](https://mlsanity.com/paper/2404.16847/) (2024)
- [MedLeak: Multimodal Medical Data Leakage in Secure Federated Learning with Crafted Models](https://mlsanity.com/paper/2407.09972/) (2024)
- [From Risk to Resilience: Towards Assessing and Mitigating the Risk of Data Reconstruction Attacks in Federated Learning](https://mlsanity.com/paper/2512.15460/) (2025)
- [Partner in Crime: Boosting Targeted Poisoning Attacks against Federated Learning](https://mlsanity.com/paper/2407.09958/) (2024)
- [Evaluating the Defense Potential of Machine Unlearning against Membership Inference Attacks](https://mlsanity.com/paper/2508.16150/) (2025)
- [Reinforcement Learning-Based Approaches for Enhancing Security and Resilience in Smart Control: A Survey on Attack and Defense Methods](https://mlsanity.com/paper/2402.15617/) (2024)
- [Federated Learning Nodes Can Reconstruct Peers' Image Data](https://mlsanity.com/paper/2410.04661/) (2024)
- [Enabling Trustworthy Federated Learning via Remote Attestation for Mitigating Byzantine Threats](https://mlsanity.com/paper/2509.00634/) (2025)
- [DictPFL: Efficient and Private Federated Learning on Encrypted Gradients](https://mlsanity.com/paper/2510.21086/) (2025)
- [No More Guessing: a Verifiable Gradient Inversion Attack in Federated Learning](https://mlsanity.com/paper/2604.15063/) (2026)
- [Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs](https://mlsanity.com/paper/2605.07961/) (2026)
- [From Efficiency to Leakage -- Privacy Backdoor in Federated Language Model Fine-Tuning](https://mlsanity.com/paper/2606.20553/) (2026)
- [Buffer is All You Need: Defending Federated Learning against Backdoor Attacks under Non-iids via Buffering](https://mlsanity.com/paper/2503.23511/) (2025)
- [On the Detectability of Active Gradient Inversion Attacks in Federated Learning](https://mlsanity.com/paper/2511.10502/) (2025)
- [FastFHE: Packing-Scalable and Depthwise-Separable CNN Inference Over FHE](https://mlsanity.com/paper/2511.22434/) (2025)
- [ARES: Scalable and Practical Gradient Inversion Attack in Federated Learning through Activation Recovery](https://mlsanity.com/paper/2603.17623/) (2026)
- [Keyed Nonlinear Transform: Lightweight Privacy-Enhancing Feature Sharing for Medical Image Analysis](https://mlsanity.com/paper/2605.14123/) (2026)
- [Cascading Gradient Inversion via LT-Code Inspired Peeling in Federated Learning](https://mlsanity.com/paper/2609.09659/) (2026)
- [Temporal Gradient Inversion for Private Trajectory Reconstruction in Embodied Reinforcement Learning](https://mlsanity.com/paper/2609.30258/) (2026)
- [Aegis: Generative Gradient Masking for Privacy-Preserving Medical Federated Learning](https://mlsanity.com/paper/2609.38339/) (2026)

## Abstract (arXiv)

Federated learning is known for its capability to safeguard the participants' data privacy. However, recently emerged model inversion attacks (MIAs) have shown that a malicious parameter server can reconstruct individual users' local data samples from model updates. The state-of-the-art attacks either rely on computation-intensive iterative optimization methods to reconstruct each input batch, making scaling difficult, or involve the malicious parameter server adding extra modules before the global model architecture, rendering the attacks too conspicuous and easily detectable. To overcome these limitations, we propose Scale-MIA, a novel MIA capable of efficiently and accurately reconstructing local training samples from the aggregated model updates, even when the system is protected by a robust secure aggregation (SA) protocol. Scale-MIA utilizes the inner architecture of models and identifies the latent space as the critical layer for breaching privacy. Scale-MIA decomposes the complex reconstruction task into an innovative two-step process. The first step is to reconstruct the latent space representations (LSRs) from the aggregated model updates using a closed-form inversion mechanism, leveraging specially crafted linear layers. Then in the second step, the LSRs are fed into a fine-tuned generative decoder to reconstruct the whole input batch. We implemented Scale-MIA on commonly used machine learning models and conducted comprehensive experiments across various settings. The results demonstrate that Scale-MIA achieves excellent performance on different datasets, exhibiting high reconstruction rates, accuracy, and attack efficiency on a larger scale compared to state-of-the-art MIAs. Our code is available at https://github.com/unknown123489/Scale-MIA.