# Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustness

arXiv: https://arxiv.org/abs/2203.13639 · HTML: https://mlsanity.com/paper/2203.13639/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 46 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)

## Built on by (most cited first, 23 of 23)

- [Certified Defences Against Adversarial Patch Attacks on Semantic Segmentation](https://mlsanity.com/paper/2209.05980/) (2022)
- [Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions](https://mlsanity.com/paper/2403.20254/) (2024)
- [PatchCURE: Improving Certifiable Robustness, Model Utility, and Computation Efficiency of Adversarial Patch Defenses](https://mlsanity.com/paper/2310.13076/) (2023)
- [JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering](https://mlsanity.com/paper/2508.05087/) (2025)
- [FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers](https://mlsanity.com/paper/2401.01752/) (2024)
- [GenFormer -- Generated Images are All You Need to Improve Robustness of Transformers on Small Datasets](https://mlsanity.com/paper/2408.14131/) (2024)
- [A Comprehensive Study on the Robustness of Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarking](https://mlsanity.com/paper/2306.12111/) (2023)
- [SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization](https://mlsanity.com/paper/2402.03317/) (2024)
- [On the interplay of adversarial robustness and architecture components: patches, convolution and attention](https://mlsanity.com/paper/2209.06953/) (2022)
- [Adversarial Attention Perturbations for Large Object Detection Transformers](https://mlsanity.com/paper/2508.02987/) (2025)
- [Attacking Transformers with Feature Diversity Adversarial Perturbation](https://mlsanity.com/paper/2403.07942/) (2024)
- [Model Science: getting serious about verification, explanation and control of AI systems](https://mlsanity.com/paper/2508.20040/) (2025)
- [Preference Redirection via Attention Concentration: An Attack on Computer Use Agents](https://mlsanity.com/paper/2604.08005/) (2026)
- [Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement](https://mlsanity.com/paper/2401.01750/) (2024)
- [Density Adaptive Attention is All You Need: Robust Parameter-Efficient Fine-Tuning Across Multiple Modalities](https://mlsanity.com/paper/2401.11143/) (2024)
- [ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer](https://mlsanity.com/paper/2409.13828/) (2024)
- [Universal and Transferable Attacks on Pathology Foundation Models](https://mlsanity.com/paper/2510.16660/) (2025)
- [Attention Deficit is Ordered! Fooling Deformable Vision Transformers with Collaborative Adversarial Patches](https://mlsanity.com/paper/2311.12914/) (2023)
- [Evaluating the Adversarial Robustness of Detection Transformers](https://mlsanity.com/paper/2412.18718/) (2024)
- [Carpet-bombing patch: attacking a deep network without usual requirements](https://mlsanity.com/paper/2212.05827/) (2022)
- [P-MSDiff: Parallel Multi-Scale Diffusion for Remote Sensing Image Segmentation](https://mlsanity.com/paper/2405.20443/) (2024)
- [Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT](https://mlsanity.com/paper/2607.07922/) (2026)
- [Corrupting Attention: Evasion-Based Adversarial Attacks on Encoder Attention in Detection Transformers](https://mlsanity.com/paper/2608.06674/) (2026)

## Abstract (arXiv)

Neural architectures based on attention such as vision transformers are revolutionizing image recognition. Their main benefit is that attention allows reasoning about all parts of a scene jointly. In this paper, we show how the global reasoning of (scaled) dot-product attention can be the source of a major vulnerability when confronted with adversarial patch attacks. We provide a theoretical understanding of this vulnerability and relate it to an adversary's ability to misdirect the attention of all queries to a single key token under the control of the adversarial patch. We propose novel adversarial objectives for crafting adversarial patches which target this vulnerability explicitly. We show the effectiveness of the proposed patch attacks on popular image classification (ViTs and DeiTs) and object detection models (DETR). We find that adversarial patches occupying 0.5% of the input can lead to robust accuracies as low as 0% for ViT on ImageNet, and reduce the mAP of DETR on MS COCO to less than 3%.