# Reveal of Vision Transformers Robustness against Adversarial Attacks

arXiv: https://arxiv.org/abs/2106.03734 · HTML: https://mlsanity.com/paper/2106.03734/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 77 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)

## Built on by (most cited first, 30 of 43)

- [How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs](https://mlsanity.com/paper/2311.16101/) (2023)
- [A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking](https://mlsanity.com/paper/2302.14301/) (2023)
- [RobustART: Benchmarking Robustness on Architecture Design and Training Techniques](https://mlsanity.com/paper/2109.05211/) (2021)
- [How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses](https://mlsanity.com/paper/2305.10862/) (2023)
- [A Light Recipe to Train Robust Vision Transformers](https://mlsanity.com/paper/2209.07399/) (2022)
- [When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture](https://mlsanity.com/paper/2210.07540/) (2022)
- [Are we ready for a new paradigm shift? A Survey on Visual Deep MLP](https://mlsanity.com/paper/2111.04060/) (2021)
- [Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs](https://mlsanity.com/paper/2110.02797/) (2021)
- [Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?](https://mlsanity.com/paper/2203.08392/) (2022)
- [Are Vision Transformers Robust to Patch Perturbations?](https://mlsanity.com/paper/2111.10659/) (2021)
- [ACTIVE: Towards Highly Transferable 3D Physical Camouflage for Universal and Robust Vehicle Evasion](https://mlsanity.com/paper/2308.07009/) (2023)
- [Towards Efficient Adversarial Training on Vision Transformers](https://mlsanity.com/paper/2207.10498/) (2022)
- [Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustness](https://mlsanity.com/paper/2203.13639/) (2022)
- [Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal](https://mlsanity.com/paper/2112.03492/) (2021)
- [MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness](https://mlsanity.com/paper/2312.04960/) (2023)
- [DBIA: Data-free Backdoor Injection Attack against Transformer Networks](https://mlsanity.com/paper/2111.11870/) (2021)
- [Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspective](https://mlsanity.com/paper/2208.09602/) (2022)
- [Curved Representation Space of Vision Transformers](https://mlsanity.com/paper/2210.05742/) (2022)
- [Finding Differences Between Transformers and ConvNets Using Counterfactual Simulation Testing](https://mlsanity.com/paper/2211.16499/) (2022)
- [Exploring Adversarial Attacks and Defenses in Vision Transformers trained with DINO](https://mlsanity.com/paper/2206.06761/) (2022)
- [Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem](https://mlsanity.com/paper/2208.00906/) (2022)
- [FG-UAP: Feature-Gathering Universal Adversarial Perturbation](https://mlsanity.com/paper/2209.13113/) (2022)
- [On Adversarial Robustness of Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2112.05871/) (2021)
- [Adversarially Pretrained Transformers May Be Universally Robust In-Context Learners](https://mlsanity.com/paper/2505.14042/) (2025)
- [Stable Vision Concept Transformers for Medical Diagnosis](https://mlsanity.com/paper/2506.05286/) (2025)
- [Mitigating Adversarial Attacks in Federated Learning with Trusted Execution Environments](https://mlsanity.com/paper/2309.07197/) (2023)
- [Vulnerability Analysis of Transformer-based Optical Character Recognition to Adversarial Attacks](https://mlsanity.com/paper/2311.17128/) (2023)
- [ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer](https://mlsanity.com/paper/2409.13828/) (2024)
- [Attacking Attention of Foundation Models Disrupts Downstream Tasks](https://mlsanity.com/paper/2506.05394/) (2025)
- [Latent Magic: An Investigation into Adversarial Examples Crafted in the Semantic Latent Space](https://mlsanity.com/paper/2305.12906/) (2023)

## Abstract (arXiv)

The major part of the vanilla vision transformer (ViT) is the attention block that brings the power of mimicking the global context of the input image. For better performance, ViT needs large-scale training data. To overcome this data hunger limitation, many ViT-based networks, or hybrid-ViT, have been proposed to include local context during the training. The robustness of ViTs and its variants against adversarial attacks has not been widely investigated in the literature like CNNs. This work studies the robustness of ViT variants 1) against different Lp-based adversarial attacks in comparison with CNNs, 2) under adversarial examples (AEs) after applying preprocessing defense methods and 3) under the adaptive attacks using expectation over transformation (EOT) framework. To that end, we run a set of experiments on 1000 images from ImageNet-1k and then provide an analysis that reveals that vanilla ViT or hybrid-ViT are more robust than CNNs. For instance, we found that 1) Vanilla ViTs or hybrid-ViTs are more robust than CNNs under Lp-based attacks and under adaptive attacks. 2) Unlike hybrid-ViTs, Vanilla ViTs are not responding to preprocessing defenses that mainly reduce the high frequency components. Furthermore, feature maps, attention maps, and Grad-CAM visualization jointly with image quality measures, and perturbations' energy spectrum are provided for an insight understanding of attention-based models.