mlsanityData to 14 Oct 2026

APRIL: Finding the Achilles' Heel on Privacy for Vision Transformers

arXiv 2112.14087 ↗First posted 28 Dec 2021cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

Federated learning frameworks typically require collaborators to share their local gradient updates of a common model instead of sharing training data to preserve privacy. However, prior works on Gradient Leakage Attacks showed that private training data can be revealed from gradients. So far almost all relevant works base their attacks on fully-connected or convolutional neural networks. Given the recent overwhelmingly rising trend of adapting Transformers to solve multifarious vision tasks, it is highly valuable to investigate the privacy risk of vision transformers. In this paper, we analyse the gradient leakage risk of self-attention based mechanism in both theoretical and practical manners. Particularly, we propose APRIL - Attention PRIvacy Leakage, which poses a strong threat to self-attention inspired models such as ViT. Showing how vision Transformers are at the risk of privacy leakage via gradients, we urge the significance of designing privacy-safer Transformer models and defending schemes.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 28 Dec 2021.

Builds on

35 of 45 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2017Non-local Neural Networkscited by 2,293
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
23 more
  1. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  2. 2019Deep Leakage from Gradientscited by 1,021
  3. 2019Towards Federated Learning at Scale: System Designcited by 832
  4. 2018Image Transformercited by 772
  5. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  6. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  7. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  8. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  9. 2020iDLG: Improved Deep Leakage from Gradientscited by 382
  10. 2019Attention Augmented Convolutional Networkscited by 363
  11. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 350
  12. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  13. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  14. 2021PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformerscited by 286
  15. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 207
  16. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  17. 2020TransPose: Keypoint Localization via Transformercited by 126
  18. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  19. 2021Gradient Inversion with Generative Image Priorcited by 101
  20. 2020R-GAP: Recursive Gradient Attack on Privacycited by 97
  21. 2019Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Recordscited by 72
  22. 2021TAG: Gradient Attack on Transformer-based Language Modelscited by 63
  23. 2021Paint Transformer: Feed Forward Neural Painting with Stroke Predictioncited by 62

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 5; 2024: 4; 2025: 5; 2026: 5.

24 papers.

Sort:

By area:

  1. 2022LAMP: Extracting Text from Gradients with Language Model Priorscited by 54
  2. 2023No One Left Behind: Real-World Federated Class-Incremental Learningcited by 53
  3. 2022A Survey on Gradient Inversion: Attacks, Defenses and Future Directionscited by 40
  4. 2023FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning Systemcited by 38
  5. 2022Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformerscited by 33
  6. 2024Poison-splat: Computation Cost Attack on 3D Gaussian Splattingcited by 18
  7. 2024DAGER: Exact Gradient Inversion for Large Language Modelscited by 14
  8. 2022Dropout is NOT All You Need to Prevent Gradient Leakagecited by 11
  9. 2023Surrogate Model Extension (SME): A Fast and Accurate Weight Update Attack on Federated Learningcited by 11
  10. 2022How Does a Deep Learning Model Architecture Impact Its Privacy? A Comprehensive Study of Privacy Attacks on CNNs and Transformerscited by 8
  11. 2023Data-Free Sketch-Based Image Retrievalcited by 7
  12. 2025Uncovering Gradient Inversion Risks in Practical Language Model Trainingcited by 7
12 more
  1. 2024MedLeak: Multimodal Medical Data Leakage in Secure Federated Learning with Crafted Modelscited by 3
  2. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  3. 2023Privacy Preserving Federated Learning with Convolutional Variational Bottleneckscited by 2
  4. 2024Dealing Doubt: Unveiling Threat Models in Gradient Inversion Attacks under Federated Learning, A Survey and Taxonomycited by 2
  5. 2025Cutting Through Privacy: A Hyperplane-Based Data Reconstruction Attack in Federated Learningcited by 2
  6. 2026Mitigating Gradient Inversion Risks in Language Models via Token Obfuscationcited by 2
  7. 2025Hear No Evil: Detecting Gradient Leakage by Malicious Servers in Federated Learningcited by 1
  8. 2026From Efficiency to Leakage -- Privacy Backdoor in Federated Language Model Fine-Tuningcited by 1
  9. 2025Evaluating Selective Encryption Against Gradient Inversion Attackscited by 0
  10. 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuningcited by 0
  11. 2026Temporal Gradient Inversion for Private Trajectory Reconstruction in Embodied Reinforcement Learningcited by 0
  12. 2026Aegis: Generative Gradient Masking for Privacy-Preserving Medical Federated Learningcited by 0