# APRIL: Finding the Achilles' Heel on Privacy for Vision Transformers

arXiv: https://arxiv.org/abs/2112.14087 · HTML: https://mlsanity.com/paper/2112.14087/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 45 references that are in the map.
- First posted on arXiv 28 Dec 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Optimization: Distributed Machine Learning for On-Device Intelligence](https://mlsanity.com/paper/1610.02527/) (2016)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [See through Gradients: Image Batch Recovery via GradInversion](https://mlsanity.com/paper/2104.07586/) (2021)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [TransPose: Keypoint Localization via Transformer](https://mlsanity.com/paper/2012.14214/) (2020)
- [MST: Masked Self-Supervised Transformer for Visual Representation](https://mlsanity.com/paper/2106.05656/) (2021)

## Built on by (most cited first, 24 of 24)

- [LAMP: Extracting Text from Gradients with Language Model Priors](https://mlsanity.com/paper/2202.08827/) (2022)
- [No One Left Behind: Real-World Federated Class-Incremental Learning](https://mlsanity.com/paper/2302.00903/) (2023)
- [A Survey on Gradient Inversion: Attacks, Defenses and Future Directions](https://mlsanity.com/paper/2206.07284/) (2022)
- [FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning System](https://mlsanity.com/paper/2303.10837/) (2023)
- [Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers](https://mlsanity.com/paper/2205.12551/) (2022)
- [Poison-splat: Computation Cost Attack on 3D Gaussian Splatting](https://mlsanity.com/paper/2410.08190/) (2024)
- [DAGER: Exact Gradient Inversion for Large Language Models](https://mlsanity.com/paper/2405.15586/) (2024)
- [Dropout is NOT All You Need to Prevent Gradient Leakage](https://mlsanity.com/paper/2208.06163/) (2022)
- [Surrogate Model Extension (SME): A Fast and Accurate Weight Update Attack on Federated Learning](https://mlsanity.com/paper/2306.00127/) (2023)
- [How Does a Deep Learning Model Architecture Impact Its Privacy? A Comprehensive Study of Privacy Attacks on CNNs and Transformers](https://mlsanity.com/paper/2210.11049/) (2022)
- [Data-Free Sketch-Based Image Retrieval](https://mlsanity.com/paper/2303.07775/) (2023)
- [Uncovering Gradient Inversion Risks in Practical Language Model Training](https://mlsanity.com/paper/2507.21198/) (2025)
- [MedLeak: Multimodal Medical Data Leakage in Secure Federated Learning with Crafted Models](https://mlsanity.com/paper/2407.09972/) (2024)
- [Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2506.04453/) (2025)
- [Privacy Preserving Federated Learning with Convolutional Variational Bottlenecks](https://mlsanity.com/paper/2309.04515/) (2023)
- [Dealing Doubt: Unveiling Threat Models in Gradient Inversion Attacks under Federated Learning, A Survey and Taxonomy](https://mlsanity.com/paper/2405.10376/) (2024)
- [Cutting Through Privacy: A Hyperplane-Based Data Reconstruction Attack in Federated Learning](https://mlsanity.com/paper/2505.10264/) (2025)
- [Mitigating Gradient Inversion Risks in Language Models via Token Obfuscation](https://mlsanity.com/paper/2602.15897/) (2026)
- [Hear No Evil: Detecting Gradient Leakage by Malicious Servers in Federated Learning](https://mlsanity.com/paper/2506.20651/) (2025)
- [From Efficiency to Leakage -- Privacy Backdoor in Federated Language Model Fine-Tuning](https://mlsanity.com/paper/2606.20553/) (2026)
- [Evaluating Selective Encryption Against Gradient Inversion Attacks](https://mlsanity.com/paper/2508.04155/) (2025)
- [MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning](https://mlsanity.com/paper/2608.01521/) (2026)
- [Temporal Gradient Inversion for Private Trajectory Reconstruction in Embodied Reinforcement Learning](https://mlsanity.com/paper/2609.30258/) (2026)
- [Aegis: Generative Gradient Masking for Privacy-Preserving Medical Federated Learning](https://mlsanity.com/paper/2609.38339/) (2026)

## Abstract (arXiv)

Federated learning frameworks typically require collaborators to share their local gradient updates of a common model instead of sharing training data to preserve privacy. However, prior works on Gradient Leakage Attacks showed that private training data can be revealed from gradients. So far almost all relevant works base their attacks on fully-connected or convolutional neural networks. Given the recent overwhelmingly rising trend of adapting Transformers to solve multifarious vision tasks, it is highly valuable to investigate the privacy risk of vision transformers. In this paper, we analyse the gradient leakage risk of self-attention based mechanism in both theoretical and practical manners. Particularly, we propose APRIL - Attention PRIvacy Leakage, which poses a strong threat to self-attention inspired models such as ViT. Showing how vision Transformers are at the risk of privacy leakage via gradients, we urge the significance of designing privacy-safer Transformer models and defending schemes.