# Seeing the Forest through the Trees: Data Leakage from Partial Transformer Gradients

arXiv: https://arxiv.org/abs/2406.00999 · HTML: https://mlsanity.com/paper/2406.00999/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [Gradient-based Adversarial Attacks against Text Transformers](https://mlsanity.com/paper/2104.13733/) (2021)
- [A Survey on Distributed Machine Learning](https://mlsanity.com/paper/1912.09789/) (2019)
- [FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare](https://mlsanity.com/paper/1907.09173/) (2019)
- [Provable Defense against Privacy Leakage in Federated Learning from Representation Perspective](https://mlsanity.com/paper/2012.06043/) (2020)
- [Recovering Private Text in Federated Learning of Language Models](https://mlsanity.com/paper/2205.08514/) (2022)
- [TAG: Gradient Attack on Transformer-based Language Models](https://mlsanity.com/paper/2103.06819/) (2021)
- [LAMP: Extracting Text from Gradients with Language Model Priors](https://mlsanity.com/paper/2202.08827/) (2022)
- [Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models](https://mlsanity.com/paper/2201.12675/) (2022)
- [Layer-wise Adaptive Model Aggregation for Scalable Federated Learning](https://mlsanity.com/paper/2110.10302/) (2021)
- [Bayesian Framework for Gradient Leakage](https://mlsanity.com/paper/2111.04706/) (2021)
- [Federated Transfer Learning: concept and applications](https://mlsanity.com/paper/2010.15561/) (2020)
- [Learning to Invert: Simple Adaptive Attacks for Gradient Inversion in Federated Learning](https://mlsanity.com/paper/2210.10880/) (2022)
- [Reconstructing Training Data from Model Gradient, Provably](https://mlsanity.com/paper/2212.03714/) (2022)
- [Revealing and Protecting Labels in Distributed Training](https://mlsanity.com/paper/2111.00556/) (2021)
- [SoK: On Gradient Leakage in Federated Learning](https://mlsanity.com/paper/2404.05403/) (2024)
- [Scalable Privacy-Preserving Distributed Learning](https://mlsanity.com/paper/2005.09532/) (2020)
- [Beyond Gradient and Priors in Privacy Attacks: Leveraging Pooler Layer Inputs of Language Models in Federated Learning](https://mlsanity.com/paper/2312.05720/) (2023)

## Built on by (most cited first, 3 of 3)

- [Empirical Calibration and Metric Differential Privacy in Language Models](https://mlsanity.com/paper/2503.13872/) (2025)
- [PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs](https://mlsanity.com/paper/2506.02965/) (2025)
- [In Transformer We Trust? A Perspective on Transformer Architecture Failure Modes](https://mlsanity.com/paper/2602.14318/) (2026)

## Abstract (arXiv)

Recent studies have shown that distributed machine learning is vulnerable to gradient inversion attacks, where private training data can be reconstructed by analyzing the gradients of the models shared in training. Previous attacks established that such reconstructions are possible using gradients from all parameters in the entire models. However, we hypothesize that most of the involved modules, or even their sub-modules, are at risk of training data leakage, and we validate such vulnerabilities in various intermediate layers of language models. Our extensive experiments reveal that gradients from a single Transformer layer, or even a single linear component with 0.54% parameters, are susceptible to training data leakage. Additionally, we show that applying differential privacy on gradients during training offers limited protection against the novel vulnerability of data disclosure.