mlsanityData to 14 Oct 2026

Intriguing Properties of Vision Transformers

arXiv 2105.10497 ↗First posted 21 May 2021cs.CV cs.AI cs.LGcited by 446

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. These models are based on multi-head self-attention mechanisms that can flexibly attend to a sequence of image patches to encode contextual cues. An important question is how such flexibility in attending image-wide context conditioned on a given patch can facilitate handling nuisances in natural images e.g., severe occlusions, domain shifts, spatial permutations, adversarial and natural perturbations. We systematically study this question via an extensive set of experiments encompassing three ViT families and comparisons with a high-performing convolutional neural network (CNN). We show and analyze the following intriguing properties of ViT: (a) Transformers are highly robust to severe occlusions, perturbations and domain shifts, e.g., retain as high as 60% top-1 accuracy on ImageNet even after randomly occluding 80% of the image content. (b) The robust performance to occlusions is not due to a bias towards local textures, and ViTs are significantly less biased towards textures compared to CNNs. When properly trained to encode shape-based features, ViTs demonstrate shape recognition capability comparable to that of human visual system, previously unmatched in the literature. (c) Using ViTs to encode shape representation leads to an interesting consequence of accurate semantic segmentation without pixel-level supervision. (d) Off-the-shelf features from a single ViT model can be combined to create a feature ensemble, leading to high accuracy rates across a range of classification datasets in both traditional and few-shot learning paradigms. We show effective features of ViTs are due to flexible and dynamic receptive fields possible via the self-attention mechanism.

Key facts. Cited by 446 papers in the map (data through 14 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 21 May 2021.

Builds on

38 of 50 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  10. 2013Intriguing properties of neural networkscited by 5,155
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
26 more
  1. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  2. 2013Describing Textures in the Wildcited by 2,125
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  5. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  6. 2021Transformers in Vision: A Surveycited by 1,013
  7. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  8. 2020Designing Network Design Spacescited by 960
  9. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  10. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  11. 2017Adversarial Patchcited by 517
  12. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  13. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  14. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 315
  15. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  16. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  17. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  18. 2019Local Relation Networks for Image Recognitioncited by 210
  19. 2021Vision Transformers are Robust Learnerscited by 207
  20. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  21. 2021Are Convolutional Neural Networks or Transformers more like human vision?cited by 109
  22. 2019Language Modeling with Deep Transformerscited by 96
  23. 2019Cross-Domain Transferability of Adversarial Perturbationscited by 92
  24. 2021A Study of Face Obfuscation in ImageNetcited by 83
  25. 2021Shape or Texture: Understanding Discriminative Features in CNNscited by 52
  26. 2021Does enhanced shape bias improve neural network robustness to common corruptions?cited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 42; 2022: 107; 2023: 111; 2024: 79; 2025: 65; 2026: 42.

446 papers.

Sort:

By area:

  1. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  2. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  3. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  4. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  5. 2022How Do Vision Transformers Work?cited by 307
  6. 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 300
  7. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  8. 2022Transformers in Medical Imaging: A Surveycited by 245
  9. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 230
  10. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  11. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  12. 2021Are Transformers More Robust Than CNNs?cited by 187
38 more
  1. 2021Partial success in closing the gap between human and machine visioncited by 185
  2. 2022Understanding The Robustness in Vision Transformerscited by 144
  3. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  4. 2021OW-DETR: Open-world Detection Transformercited by 129
  5. 2023MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformercited by 124
  6. 20223D Common Corruptions and Data Augmentationcited by 119
  7. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  8. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  9. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  10. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  11. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 108
  12. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 95
  13. 2022Vision Transformers provably learn spatial structurecited by 93
  14. 2021Delving Deep into the Generalization of Vision Transformers under Distribution Shiftscited by 85
  15. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  16. 2021RobustART: Benchmarking Robustness on Architecture Design and Training Techniquescited by 79
  17. 2021TransMix: Attend to Mix for Vision Transformerscited by 79
  18. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  19. 2022What do Vision Transformers Learn? A Visual Explorationcited by 76
  20. 2023Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-traincited by 75
  21. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  22. 2024Wavelet Convolutions for Large Receptive Fieldscited by 74
  23. 2021On Improving Adversarial Transferability of Vision Transformerscited by 70
  24. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 66
  25. 2022Ghost-free High Dynamic Range Imaging with Context-aware Transformercited by 65
  26. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  27. 2021Self-supervised Video Transformercited by 64
  28. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  29. 2022Video Transformers: A Surveycited by 63
  30. 2022Transformers in Remote Sensing: A Surveycited by 61
  31. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 59
  32. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  33. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  34. 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
  35. 2022MARLIN: Masked Autoencoder for facial video Representation LearnINgcited by 57
  36. 2022Stepwise Feature Fusion: Local Guides Globalcited by 56
  37. 2022Unsupervised Salient Object Detection with Spectral Cluster Votingcited by 56
  38. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56

50 most cited of 446 All 446 (JSON)