mlsanityData to 14 Oct 2026

Are Vision Transformers Robust to Patch Perturbations?

arXiv 2111.10659 ↗First posted 20 Nov 2021cs.CV eess.IVcited by 39

Not on the map. Why?

Abstract (arXiv)

Recent advances in Vision Transformer (ViT) have demonstrated its impressive performance in image classification, which makes it a promising alternative to Convolutional Neural Network (CNN). Unlike CNNs, ViT represents an input image as a sequence of image patches. The patch-based input image representation makes the following question interesting: How does ViT perform when individual input image patches are perturbed with natural corruptions or adversarial perturbations, compared to CNNs? In this work, we study the robustness of ViT to patch-wise perturbations. Surprisingly, we find that ViTs are more robust to naturally corrupted patches than CNNs, whereas they are more vulnerable to adversarial patches. Furthermore, we discover that the attention mechanism greatly affects the robustness of vision transformers. Specifically, the attention module can help improve the robustness of ViT by effectively ignoring natural corrupted patches. However, when ViTs are attacked by an adversary, the attention mechanism can be easily fooled to focus more on the adversarially perturbed patches and cause a mistake. Based on our analysis, we propose a simple temperature-scaling based method to improve the robustness of ViT against adversarial patches. Extensive qualitative and quantitative experiments are performed to support our findings, understanding, and improvement of ViT robustness to patch-wise perturbations across a set of transformer-based architectures.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 20 Nov 2021.

Builds on

45 of 59 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2013Intriguing properties of neural networkscited by 5,155
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  11. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  12. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
33 more
  1. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  3. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  4. 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
  5. 2020Quantifying Attention Flow in Transformerscited by 775
  6. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  7. 2021Transformer in Transformercited by 633
  8. 2017Adversarial Patchcited by 517
  9. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  10. 2021Intriguing Properties of Vision Transformerscited by 446
  11. 2021Early Convolutions Help Transformers See Bettercited by 396
  12. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  13. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 334
  14. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  15. 2021Vision Transformers are Robust Learnerscited by 207
  16. 2021Are Transformers More Robust Than CNNs?cited by 187
  17. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  18. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
  19. 2021Visformer: The Vision-friendly Transformercited by 137
  20. 2021Towards Robust Vision Transformercited by 137
  21. 2018LaVAN: Localized and Visible Adversarial Noisecited by 113
  22. 2021RobustART: Benchmarking Robustness on Architecture Design and Training Techniquescited by 79
  23. 2021On Improving Adversarial Transferability of Vision Transformerscited by 70
  24. 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
  25. 2021Certified Patch Robustness via Smoothed Vision Transformerscited by 45
  26. 2020CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generationcited by 44
  27. 2021Reveal of Vision Transformers Robustness against Adversarial Attackscited by 43
  28. 2018Understanding Individual Decisions of CNNs via Contrastive Backpropagationcited by 40
  29. 2021Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentationcited by 40
  30. 2020Improving Calibration through the Relationship with Adversarial Robustnesscited by 26
  31. 2021Meta Adversarial Training against Universal Patchescited by 19
  32. 2019Saliency Methods for Explaining Adversarial Attackscited by 16
  33. 2021Adversarial Token Attacks on Vision Transformerscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 16; 2024: 9; 2025: 5; 2026: 5.

39 papers.

Sort:

By area:

  1. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  2. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Imagescited by 69
  3. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 66
  4. 2023Backdoor Defense via Adaptively Splitting Poisoned Datasetcited by 43
  5. 2024Revisiting Adversarial Training at Scalecited by 33
  6. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 31
  7. 2023Robustness of SAM: Segment Anything Under Corruptions and Beyondcited by 31
  8. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 30
  9. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Modelcited by 22
  10. 2023A Survey on the Robustness of Computer Vision Models against Common Corruptionscited by 21
  11. 2023Robustifying Token Attention for Vision Transformerscited by 19
  12. 2024Which Model Generated This Image? A Model-Agnostic Approach for Origin Attributioncited by 16
27 more
  1. 2023Reliable Evaluation of Adversarial Transferabilitycited by 13
  2. 2023MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustnesscited by 12
  3. 2022Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspectivecited by 8
  4. 2022Towards Improved Input Masking for Convolutional Neural Networkscited by 8
  5. 2023Towards Robust Prompts on Vision-Language Modelscited by 7
  6. 2023Fairness-aware Vision Transformer via Debiased Self-Attentioncited by 6
  7. 2024Learning Visual Prompts for Guiding the Attention of Vision Transformerscited by 6
  8. 2022On the interplay of adversarial robustness and architecture components: patches, convolution and attentioncited by 4
  9. 2023A Study on the Generality of Neural Network Structures for Monocular Depth Estimationcited by 4
  10. 2025Benchmarking the Spatial Robustness of DNNs via Natural and Adversarial Localized Corruptionscited by 3
  11. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Modelscited by 3
  12. 2023Random Position Adversarial Patch for Vision Transformerscited by 2
  13. 2023FLARE: Fingerprinting Deep Reinforcement Learning Agents using Universal Adversarial Maskscited by 2
  14. 2024Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinementcited by 2
  15. 2024ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformercited by 2
  16. 2024SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformerscited by 2
  17. 2025Attacking Attention of Foundation Models Disrupts Downstream Taskscited by 2
  18. 2025Universal and Transferable Attacks on Pathology Foundation Modelscited by 2
  19. 2023Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attackscited by 1
  20. 2023Explainability and Robustness of Deep Visual Classification Modelscited by 0
  21. 2024Adversarial Robustness for Deep Learning-based Wildfire Prediction Modelscited by 0
  22. 2025Vision Transformers: the threat of realistic adversarial patchescited by 0
  23. 2025Adversarially Robust Few-Shot Anomaly Detection with Vision Foundation Modelscited by 0
  24. 2026KidVis: Do Multimodal Large Language Models Possess the Visual Perceptual Capabilities of a 6-Year-Old?cited by 0
  25. 2026Enhancing Few-Shot Out-of-Distribution Detection via the Refinement of Foreground and Backgroundcited by 0
  26. 2026Adversarial Decoys: Misdirecting Attention-Based Defenses in ViTcited by 0
  27. 2026DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detectioncited by 0