mlsanityData to 7 Oct 2026

Understanding Robustness of Transformers for Image Classification

arXiv 2103.14586 ↗First posted 26 Mar 2021cs.CV cs.AI cs.LGcited by 250

Not on the map. Why?

Abstract (arXiv)

Deep Convolutional Neural Networks (CNNs) have long been the architecture of choice for computer vision tasks. Recently, Transformer-based architectures like Vision Transformer (ViT) have matched or even surpassed ResNets for image classification. However, details of the Transformer architecture -- such as the use of non-overlapping patches -- lead one to wonder whether these networks are as robust. In this paper, we perform an extensive study of a variety of different measures of robustness of ViT models and compare the findings to ResNet baselines. We investigate robustness to input perturbations as well as robustness to model perturbations. We find that when pre-trained with a sufficient amount of data, ViT models are at least as robust as the ResNet counterparts on a broad range of perturbations. We also find that Transformers are robust to the removal of almost any single layer, and that while activations from later layers are highly correlated with each other, they nevertheless play an important role in classification.

Key facts. Cited by 250 papers in the map (data through 7 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 26 Mar 2021.

Builds on

46 of 55 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
34 more
  1. 2013Intriguing properties of neural networkscited by 5,114
  2. 2017Squeeze-and-Excitation Networkscited by 4,763
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  15. 2021Transformers in Vision: A Surveycited by 1,009
  16. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  17. 2020Object-Centric Learning with Slot Attentioncited by 865
  18. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  19. 2019Are Sixteen Heads Really Better than One?cited by 803
  20. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  21. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  22. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  23. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  24. 2020Exploring Self-attention for Image Recognitioncited by 313
  25. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  26. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  27. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  28. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  29. 2016Highway and Residual Networks learn Unrolled Iterative Estimationcited by 52
  30. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  31. 2014Deformable Part Models are Convolutional Neural Networkscited by 35
  32. 2019Are Adversarial Robustness and Common Perturbation Robustness Independent Attributes ?cited by 25
  33. 2020Pay Attention when Requiredcited by 8
  34. 2020Adversarial and Natural Perturbations for General Robustnesscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 66; 2023: 63; 2024: 49; 2025: 28; 2026: 15.

250 papers.

Sort:

By area:

  1. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  2. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  3. 2021Intriguing Properties of Vision Transformerscited by 446
  4. 2022How Do Vision Transformers Work?cited by 305
  5. 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 299
  6. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  7. 2022Transformers in Medical Imaging: A Surveycited by 245
  8. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  9. 2021Vision Transformers are Robust Learnerscited by 208
  10. 2021Are Transformers More Robust Than CNNs?cited by 187
  11. 2021Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformerscited by 159
  12. 2022FlexiViT: One Model for All Patch Sizescited by 151
38 more
  1. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  2. 2021Towards Robust Vision Transformercited by 137
  3. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  4. 20223D Common Corruptions and Data Augmentationcited by 120
  5. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  6. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  7. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  8. 20223D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentationcited by 94
  9. 2022Vision Transformers provably learn spatial structurecited by 92
  10. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  11. 2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
  12. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  13. 2023A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinkingcited by 85
  14. 2022Human alignment of neural network representationscited by 74
  15. 2021On Improving Adversarial Transferability of Vision Transformerscited by 70
  16. 2022Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstructioncited by 67
  17. 2022Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imagingcited by 66
  18. 2022A Light Recipe to Train Robust Vision Transformerscited by 64
  19. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  20. 2022Video Transformers: A Surveycited by 63
  21. 2022Flow-Guided Transformer for Video Inpaintingcited by 59
  22. 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
  23. 2022A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributescited by 56
  24. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  25. 2024Transformer Layers as Painterscited by 50
  26. 2022ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpointscited by 48
  27. 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 47
  28. 2021OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Imagescited by 46
  29. 2021Reveal of Vision Transformers Robustness against Adversarial Attackscited by 43
  30. 2023Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularizationcited by 42
  31. 2021Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentationcited by 40
  32. 2021PRIME: A few primitives can boost robustness to common corruptionscited by 40
  33. 2021Are Vision Transformers Robust to Patch Perturbations?cited by 39
  34. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 38
  35. 2023Understanding the Robustness of 3D Object Detection with Bird's-Eye-View Representations in Autonomous Drivingcited by 37
  36. 2022Are Vision Transformers Robust to Spurious Correlations?cited by 36
  37. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  38. 2021Discrete Representations Strengthen Vision Transformer Robustnesscited by 34

50 most cited of 250 All 250 (JSON)