mlsanityData to 7 Oct 2026

How Do Vision Transformers Work?

arXiv 2202.06709 ↗First posted 14 Feb 2022cs.CV cs.LGcited by 305

Not on the map. Why?

Abstract (arXiv)

The success of multi-head self-attentions (MSAs) for computer vision is now indisputable. However, little is known about how MSAs work. We present fundamental explanations to help better understand the nature of MSAs. In particular, we demonstrate the following properties of MSAs and Vision Transformers (ViTs): (1) MSAs improve not only accuracy but also generalization by flattening the loss landscapes. Such improvement is primarily attributable to their data specificity, not long-range dependency. On the other hand, ViTs suffer from non-convex losses. Large datasets and loss landscape smoothing methods alleviate this problem; (2) MSAs and Convs exhibit opposite behaviors. For example, MSAs are low-pass filters, but Convs are high-pass filters. Therefore, MSAs and Convs are complementary; (3) Multi-stage neural networks behave like a series connection of small individual models. In addition, MSAs at the end of a stage play a key role in prediction. Based on these insights, we propose AlterNet, a model in which Conv blocks at the end of a stage are replaced with MSA blocks. AlterNet outperforms CNNs not only in large data regimes but also in small data regimes. The code is available at https://github.com/xxxnell/how-do-vits-work.

Key facts. Cited by 305 papers in the map (data through 7 Oct 2026). Builds on 68 of its 77 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

68 of 77 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
38 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  3. 2017Non-local Neural Networkscited by 2,290
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  8. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2017Random Erasing Data Augmentationcited by 1,235
  12. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  13. 2016Deep Networks with Stochastic Depthcited by 973
  14. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  15. 2019Are Sixteen Heads Really Better than One?cited by 803
  16. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  17. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  18. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  19. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  20. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  21. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  22. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  23. 2021Intriguing Properties of Vision Transformerscited by 446
  24. 2021Early Convolutions Help Transformers See Bettercited by 395
  25. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  26. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  27. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  28. 2019Attention Augmented Convolutional Networkscited by 363
  29. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  30. 2018How Does Batch Normalization Help Optimization?cited by 355
  31. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  32. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  33. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  34. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  35. 2019A Fourier Perspective on Model Robustness in Computer Visioncited by 288
  36. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  37. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  38. 2021Understanding Robustness of Transformers for Image Classificationcited by 250

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 68; 2023: 83; 2024: 72; 2025: 45; 2026: 37.

305 papers.

Sort:

By area:

  1. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  2. 2022Vision Transformers for Single Image Dehazingcited by 255
  3. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  4. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  5. 2023Understanding plasticity in neural networkscited by 170
  6. 2023Graph Inductive Biases in Transformers without Message Passingcited by 150
  7. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  8. 2022Inception Transformercited by 99
  9. 2024CATCH: Channel-Aware multivariate Time Series Anomaly Detection via Frequency Patchingcited by 91
  10. 2023What Do Self-Supervised Vision Transformers Learn?cited by 90
  11. 2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
  12. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
38 more
  1. 2022CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flowcited by 82
  2. 2023MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor Formula for Image Dehazingcited by 81
  3. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  4. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 75
  5. 2022What do Vision Transformers Learn? A Visual Explorationcited by 75
  6. 2024Wavelet Convolutions for Large Receptive Fieldscited by 74
  7. 2023Semantic Segmentation using Vision Transformers: A surveycited by 73
  8. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  9. 2022Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarioscited by 65
  10. 2023Token Contrast for Weakly-Supervised Semantic Segmentationcited by 64
  11. 2023PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networkscited by 62
  12. 2022Transformers in Remote Sensing: A Surveycited by 61
  13. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  14. 2023Frequency-Aware Transformer for Learned Image Compressioncited by 56
  15. 2024SHViT: Single-Head Vision Transformer with Memory Efficient Macro Designcited by 55
  16. 2025DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generationcited by 55
  17. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 53
  18. 2023AttentionViz: A Global View of Transformer Attentioncited by 50
  19. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  20. 2023Masked Image Modeling with Local Multi-Scale Reconstructioncited by 43
  21. 2024EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Cameracited by 42
  22. 2022The 8-Point Algorithm as an Inductive Bias for Relative Pose Prediction by ViTscited by 38
  23. 2022Natural Color Fool: Towards Boosting Black-box Unrestricted Attackscited by 38
  24. 2022Are Vision Transformers Robust to Spurious Correlations?cited by 36
  25. 2022Rarity Score : A New Metric to Evaluate the Uncommonness of Synthesized Imagescited by 35
  26. 2022How to Train Vision Transformer on Small-scale Datasets?cited by 34
  27. 2023Multi-Architecture Multi-Expert Diffusion Modelscited by 34
  28. 2024Frequency-Adaptive Dilated Convolution for Semantic Segmentationcited by 34
  29. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  30. 2023Rethinking Optical Flow from Geometric Matching Consistent Perspectivecited by 33
  31. 2022Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectivescited by 32
  32. 2022Architecture-Agnostic Masked Image Modeling -- From ViT back to CNNcited by 31
  33. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
  34. 2023LGViT: Dynamic Early Exiting for Accelerating Vision Transformercited by 30
  35. 2023Koopman neural operator as a mesh-free solver of non-linear partial differential equationscited by 29
  36. 2022Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasetscited by 28
  37. 2023ALOFT: A Lightweight MLP-like Architecture with Dynamic Low-frequency Transform for Domain Generalizationcited by 28
  38. 2023Diverse Cotraining Makes Strong Semi-Supervised Segmentorcited by 26

50 most cited of 305 All 305 (JSON)