mlsanityData to 7 Oct 2026

Adversarial Token Attacks on Vision Transformers

arXiv 2110.04337 ↗First posted 8 Oct 2021cs.CV cs.CR cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Vision transformers rely on a patch token based self attention mechanism, in contrast to convolutional networks. We investigate fundamental differences between these two families of models, by designing a block sparsity based adversarial token attack. We probe and analyze transformer as well as convolutional models with token attacks of varying patch sizes. We infer that transformer models are more sensitive to token attacks than convolutional models, with ResNets outperforming Transformer models by up to ~30% in robust accuracy for single token attacks.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 52 of its 59 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

52 of 59 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2016Wide Residual Networkscited by 3,057
38 more
  1. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  2. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  3. 2016Adversarial examples in the physical worldcited by 1,762
  4. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  6. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  8. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  9. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  10. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  11. 2018Image Transformercited by 770
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  13. 2021Going deeper with Image Transformerscited by 759
  14. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  15. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  16. 2017Adversarial Patchcited by 514
  17. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  18. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  19. 2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
  20. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 328
  21. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  22. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 275
  23. 2019Fooling automated surveillance cameras: adversarial patches to attack person detectioncited by 268
  24. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  25. 2021Vision Transformers are Robust Learnerscited by 208
  26. 2020Bag of Tricks for Adversarial Trainingcited by 199
  27. 2018Spatially Transformed Adversarial Examplescited by 195
  28. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  29. 2019Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectorscited by 155
  30. 2018Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priorscited by 145
  31. 2018Constructing Unrestricted Adversarial Examples with Generative Modelscited by 135
  32. 2019Sparse and Imperceivable Adversarial Attackscited by 95
  33. 2017Exploring the Landscape of Spatial Robustnesscited by 81
  34. 2020Sparse-RS: a versatile framework for query-efficient sparse black-box adversarial attackscited by 65
  35. 2020The Translucent Patch: A Physical and Universal Attack on Object Detectorscited by 62
  36. 2019Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifierscited by 56
  37. 2018Maximal Jacobian-based Saliency Map Attackcited by 46
  38. 2018Defense Against the Dark Arts: An overview of adversarial example security research and future research directionscited by 11

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 0; 2024: 1; 2025: 2; 2026: 1.

11 papers.

  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2021Are Vision Transformers Robust to Patch Perturbations?cited by 39
  4. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 29
  5. 2022Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustnesscited by 23
  6. 2022Defending Backdoor Attacks on Vision Transformer via Patch Processingcited by 19
  7. 2025Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Modelscited by 9
  8. 2021DBIA: Data-free Backdoor Injection Attack against Transformer Networkscited by 8
  9. 2022Revealing Secrets From Pre-trained Modelscited by 0
  10. 2024Multi-Attribute Vision Transformers are Efficient and Robust Learnerscited by 0
  11. 2026TEE-X: TEE-aware Acceleration Framework for Large Vision Models at the Edgecited by 0