# Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs

arXiv: https://arxiv.org/abs/2110.02797 · HTML: https://mlsanity.com/paper/2110.02797/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 83 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 30 of 73)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)

## Built on by (most cited first, 30 of 57)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs](https://mlsanity.com/paper/2311.16101/) (2023)
- [A Light Recipe to Train Robust Vision Transformers](https://mlsanity.com/paper/2209.07399/) (2022)
- [When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture](https://mlsanity.com/paper/2210.07540/) (2022)
- [Are we ready for a new paradigm shift? A Survey on Visual Deep MLP](https://mlsanity.com/paper/2111.04060/) (2021)
- [A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes](https://mlsanity.com/paper/2201.10766/) (2022)
- [Are Vision Transformers Robust to Patch Perturbations?](https://mlsanity.com/paper/2111.10659/) (2021)
- [Robustness of SAM: Segment Anything Under Corruptions and Beyond](https://mlsanity.com/paper/2306.07713/) (2023)
- [ACTIVE: Towards Highly Transferable 3D Physical Camouflage for Universal and Robust Vehicle Evasion](https://mlsanity.com/paper/2308.07009/) (2023)
- [Towards Efficient Adversarial Training on Vision Transformers](https://mlsanity.com/paper/2207.10498/) (2022)
- [Investigating Top-k White-Box and Transferable Black-box Attack](https://mlsanity.com/paper/2204.00089/) (2022)
- [Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustness](https://mlsanity.com/paper/2203.13639/) (2022)
- [Defending Backdoor Attacks on Vision Transformer via Patch Processing](https://mlsanity.com/paper/2206.12381/) (2022)
- [Robustifying Token Attention for Vision Transformers](https://mlsanity.com/paper/2303.11126/) (2023)
- [Reliable Evaluation of Adversarial Transferability](https://mlsanity.com/paper/2306.08565/) (2023)
- [Robust Transformer with Locality Inductive Bias and Feature Normalization](https://mlsanity.com/paper/2301.11553/) (2023)
- [HybridAugment++: Unified Frequency Spectra Perturbations for Model Robustness](https://mlsanity.com/paper/2307.11823/) (2023)
- [MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness](https://mlsanity.com/paper/2312.04960/) (2023)
- [Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions](https://mlsanity.com/paper/2403.20254/) (2024)
- [Exploring the Adversarial Robustness of CLIP for AI-generated Image Detection](https://mlsanity.com/paper/2407.19553/) (2024)
- [Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspective](https://mlsanity.com/paper/2208.09602/) (2022)
- [Curved Representation Space of Vision Transformers](https://mlsanity.com/paper/2210.05742/) (2022)
- [On the Effectiveness of Spectral Discriminators for Perceptual Quality Improvement](https://mlsanity.com/paper/2307.12027/) (2023)
- [CLAD: A Contrastive Learning based Approach for Background Debiasing](https://mlsanity.com/paper/2210.02748/) (2022)
- [Finding Differences Between Transformers and ConvNets Using Counterfactual Simulation Testing](https://mlsanity.com/paper/2211.16499/) (2022)
- [CertViT: Certified Robustness of Pre-Trained Vision Transformers](https://mlsanity.com/paper/2302.10287/) (2023)
- [Attack-SAM: Towards Attacking Segment Anything Model With Adversarial Examples](https://mlsanity.com/paper/2305.00866/) (2023)
- [Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem](https://mlsanity.com/paper/2208.00906/) (2022)
- [Empirical Advocacy of Bio-inspired Models for Robust Image Recognition](https://mlsanity.com/paper/2205.09037/) (2022)
- [Toward Robust Diagnosis: A Contour Attention Preserving Adversarial Defense for COVID-19 Detection](https://mlsanity.com/paper/2211.16806/) (2022)

## Abstract (arXiv)

Convolutional Neural Networks (CNNs) have become the de facto gold standard in computer vision applications in the past years. Recently, however, new model architectures have been proposed challenging the status quo. The Vision Transformer (ViT) relies solely on attention modules, while the MLP-Mixer architecture substitutes the self-attention modules with Multi-Layer Perceptrons (MLPs). Despite their great success, CNNs have been widely known to be vulnerable to adversarial attacks, causing serious concerns for security-sensitive applications. Thus, it is critical for the community to know whether the newly proposed ViT and MLP-Mixer are also vulnerable to adversarial attacks. To this end, we empirically evaluate their adversarial robustness under several adversarial attack setups and benchmark them against the widely used CNNs. Overall, we find that the two architectures, especially ViT, are more robust than their CNN models. Using a toy example, we also provide empirical evidence that the lower adversarial robustness of CNNs can be partially attributed to their shift-invariant property. Our frequency analysis suggests that the most robust ViT architectures tend to rely more on low-frequency features compared with CNNs. Additionally, we have an intriguing finding that MLP-Mixer is extremely vulnerable to universal adversarial perturbations.