mlsanityData to 14 Oct 2026

Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustness

arXiv 2203.13639 ↗First posted 25 Mar 2022cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Neural architectures based on attention such as vision transformers are revolutionizing image recognition. Their main benefit is that attention allows reasoning about all parts of a scene jointly. In this paper, we show how the global reasoning of (scaled) dot-product attention can be the source of a major vulnerability when confronted with adversarial patch attacks. We provide a theoretical understanding of this vulnerability and relate it to an adversary's ability to misdirect the attention of all queries to a single key token under the control of the adversarial patch. We propose novel adversarial objectives for crafting adversarial patches which target this vulnerability explicitly. We show the effectiveness of the proposed patch attacks on popular image classification (ViTs and DeiTs) and object detection models (DETR). We find that adversarial patches occupying 0.5% of the input can lead to robust accuracies as low as 0% for ViT on ImageNet, and reduce the mAP of DETR on MS COCO to less than 3%.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 42 of its 46 references that are in the map. First posted on arXiv 25 Mar 2022.

Builds on

42 of 46 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  12. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,887
30 more
  1. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  2. 2021ViViT: A Video Vision Transformercited by 1,620
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  4. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  5. 2021Video Swin Transformercited by 1,091
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  7. 2020Pre-Trained Image Processing Transformercited by 893
  8. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 866
  9. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  10. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  11. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  12. 2021Intriguing Properties of Vision Transformerscited by 446
  13. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  14. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 347
  15. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  16. 2018Video Action Transformer Networkcited by 316
  17. 2021Video Transformer Networkcited by 272
  18. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  19. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  20. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  21. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 158
  22. 2020Toward Transformer-Based Object Detectioncited by 84
  23. 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
  24. 2021On Improving Adversarial Transferability of Vision Transformerscited by 70
  25. 2020Sparse-RS: a versatile framework for query-efficient sparse black-box adversarial attackscited by 65
  26. 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
  27. 2020Parameter Efficient Multimodal Transformers for Video Representation Learningcited by 52
  28. 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 49
  29. 2021Reveal of Vision Transformers Robustness against Adversarial Attackscited by 43
  30. 2021Adversarial Token Attacks on Vision Transformerscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 10; 2025: 4; 2026: 3.

23 papers.

Sort:

By area:

  1. 2022Certified Defences Against Adversarial Patch Attacks on Semantic Segmentationcited by 16
  2. 2024Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptionscited by 12
  3. 2023PatchCURE: Improving Certifiable Robustness, Model Utility, and Computation Efficiency of Adversarial Patch Defensescited by 11
  4. 2025JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steeringcited by 11
  5. 2024FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformerscited by 9
  6. 2024GenFormer -- Generated Images are All You Need to Improve Robustness of Transformers on Small Datasetscited by 6
  7. 2023A Comprehensive Study on the Robustness of Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarkingcited by 5
  8. 2024SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalizationcited by 5
  9. 2022On the interplay of adversarial robustness and architecture components: patches, convolution and attentioncited by 4
  10. 2025Adversarial Attention Perturbations for Large Object Detection Transformerscited by 4
  11. 2024Attacking Transformers with Feature Diversity Adversarial Perturbationcited by 3
  12. 2025Model Science: getting serious about verification, explanation and control of AI systemscited by 3
11 more
  1. 2026Preference Redirection via Attention Concentration: An Attack on Computer Use Agentscited by 3
  2. 2024Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinementcited by 2
  3. 2024Density Adaptive Attention is All You Need: Robust Parameter-Efficient Fine-Tuning Across Multiple Modalitiescited by 2
  4. 2024ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformercited by 2
  5. 2025Universal and Transferable Attacks on Pathology Foundation Modelscited by 2
  6. 2023Attention Deficit is Ordered! Fooling Deformable Vision Transformers with Collaborative Adversarial Patchescited by 1
  7. 2024Evaluating the Adversarial Robustness of Detection Transformerscited by 1
  8. 2022Carpet-bombing patch: attacking a deep network without usual requirementscited by 0
  9. 2024P-MSDiff: Parallel Multi-Scale Diffusion for Remote Sensing Image Segmentationcited by 0
  10. 2026Adversarial Decoys: Misdirecting Attention-Based Defenses in ViTcited by 0
  11. 2026Corrupting Attention: Evasion-Based Adversarial Attacks on Encoder Attention in Detection Transformerscited by 0