mlsanityData to 14 Oct 2026

Towards Transferable Adversarial Attacks on Vision Transformers

arXiv 2109.04176 ↗First posted 9 Sep 2021cs.CV cs.AIcited by 80

Not on the map. Why?

Abstract (arXiv)

Vision transformers (ViTs) have demonstrated impressive performance on a series of computer vision tasks, yet they still suffer from adversarial examples. % crafted in a similar fashion as CNNs. In this paper, we posit that adversarial attacks on transformers should be specially tailored for their architecture, jointly considering both patches and self-attention, in order to achieve high transferability. More specifically, we introduce a dual attack framework, which contains a Pay No Attention (PNA) attack and a PatchOut attack, to improve the transferability of adversarial samples across different ViTs. We show that skipping the gradients of attention during backpropagation can generate adversarial examples with high transferability. In addition, adversarial perturbations generated by optimizing randomly sampled subsets of patches at each iteration achieve higher attack success rates than attacks using all patches. We evaluate the transferability of attacks on state-of-the-art ViTs, CNNs and robustly trained CNNs. The results of these experiments demonstrate that the proposed dual attack can greatly boost transferability between ViTs and from ViTs to CNNs. In addition, the proposed method can easily be combined with existing transfer methods to boost performance. Code is available at https://github.com/zhipeng-wei/PNA-PatchOut.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

32 of 44 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  9. 2016Adversarial examples in the physical worldcited by 1,769
  10. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  12. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
20 more
  1. 2016Adversarial Machine Learning at Scalecited by 936
  2. 2021Going deeper with Image Transformerscited by 764
  3. 2021Transformer in Transformercited by 633
  4. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  5. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  6. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  7. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  8. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 442
  9. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  10. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 355
  11. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  12. 2021Vision Transformers are Robust Learnerscited by 207
  13. 2020Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNetscited by 167
  14. 2021Visformer: The Vision-friendly Transformercited by 137
  15. 2021Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical Worldcited by 127
  16. 2021M2TR: Multi-modal Multi-scale Transformers for Deepfake Detectioncited by 114
  17. 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
  18. 2021On Improving Adversarial Transferability of Vision Transformerscited by 70
  19. 2019Heuristic Black-box Adversarial Attacks on Video Recognition Modelscited by 37
  20. 2020Sparse Black-box Video Attack with Reinforcement Learningcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 10; 2023: 17; 2024: 17; 2025: 21; 2026: 9.

80 papers.

Sort:

By area:

  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  3. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 108
  4. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  5. 2022Physical Adversarial Attack meets Computer Vision: A Decade Surveycited by 67
  6. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 66
  7. 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
  8. 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 49
  9. 2023Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularizationcited by 42
  10. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentationcited by 39
  11. 2022Enhancing the Self-Universality for Transferable Targeted Attackscited by 30
  12. 2022Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustnesscited by 23
38 more
  1. 2022Exploring the Relationship between Architecture and Adversarially Robust Generalizationcited by 23
  2. 2022Towards Comprehensive Testing on the Robustness of Cooperative Multi-agent Reinforcement Learningcited by 21
  3. 2024Boosting Adversarial Transferability across Model Genus by Deformation-Constrained Warpingcited by 18
  4. 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMscited by 18
  5. 2021Attacking Video Recognition Models with Bullet-Screen Commentscited by 16
  6. 2021Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removalcited by 16
  7. 2023Reliable Evaluation of Adversarial Transferabilitycited by 13
  8. 2023Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Promptingcited by 12
  9. 2023Vulnerabilities in Video Quality Assessment Models: The Challenge of Adversarial Attackscited by 12
  10. 2024On the Robustness of Kolmogorov-Arnold Networks: An Adversarial Perspectivecited by 11
  11. 2023Improving Adversarial Transferability via Model Alignmentcited by 10
  12. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  13. 2025Cross-Modal Transferable Image-to-Video Attack on Video Quality Metricscited by 9
  14. 2023Boosting Adversarial Transferability with Learnable Patch-wise Maskscited by 8
  15. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 8
  16. 2025Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinementcited by 7
  17. 2023Towards Evaluating Transfer-based Attacks Systematically, Practically, and Fairlycited by 6
  18. 2022Long-term Leap Attention, Short-term Periodic Shift for Video Classificationcited by 5
  19. 2024Multimodal Attack Detection for Action Recognition Modelscited by 5
  20. 2025Benchmarking Unified Face Attack Detection via Hierarchical Prompt Tuningcited by 5
  21. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  22. 2021Burn After Reading: Online Adaptation for Cross-domain Streaming Datacited by 3
  23. 2021On Adversarial Robustness of Point Cloud Semantic Segmentationcited by 3
  24. 2024Attacking Transformers with Feature Diversity Adversarial Perturbationcited by 3
  25. 2025Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Featurescited by 3
  26. 2023Vulnerability Analysis of Transformer-based Optical Character Recognition to Adversarial Attackscited by 2
  27. 2023QuantAttack: Exploiting Dynamic Quantization to Attack Vision Transformerscited by 2
  28. 2024DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms in Vision Transformerscited by 2
  29. 2024Query-Efficient Hard-Label Black-Box Attack against Vision Transformerscited by 2
  30. 2024ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformercited by 2
  31. 2024Enhancing Adversarial Robustness via Uncertainty-Aware Distributional Adversarial Trainingcited by 2
  32. 2025TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularizationcited by 2
  33. 2025Attacking Attention of Foundation Models Disrupts Downstream Taskscited by 2
  34. 2025Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformerscited by 2
  35. 2025Universal and Transferable Attacks on Pathology Foundation Modelscited by 2
  36. 2025Boosting Adversarial Transferability via Ensemble Non-Attentioncited by 2
  37. 2023Common Knowledge Learning for Generating Transferable Adversarial Examplescited by 1
  38. 2023Attention Deficit is Ordered! Fooling Deformable Vision Transformers with Collaborative Adversarial Patchescited by 1

50 most cited of 80 All 80 (JSON)