mlsanityData to 7 Oct 2026

Three things everyone should know about Vision Transformers

arXiv 2203.09795 ↗First posted 18 Mar 2022cs.CVcited by 90

Not on the map. Why?

Abstract (arXiv)

After their initial success in natural language processing, transformer architectures have rapidly gained traction in computer vision, providing state-of-the-art results for tasks such as image classification, detection, segmentation, and video analysis. We offer three insights based on simple and easy to implement variants of vision transformers. (1) The residual layers of vision transformers, which are usually processed sequentially, can to some extent be processed efficiently in parallel without noticeably affecting the accuracy. (2) Fine-tuning the weights of the attention layers is sufficient to adapt vision transformers to a higher resolution and to other classification tasks. This saves compute, reduces the peak memory consumption at fine-tuning time, and allows sharing the majority of weights across tasks. (3) Adding MLP-based patch pre-processing layers improves Bert-like self-supervised training based on patch masking. We evaluate the impact of these design choices using the ImageNet-1k dataset, and confirm our findings on the ImageNet-v2 test set. Transfer performance is measured across six smaller datasets.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 69 of its 76 references that are in the map. First posted on arXiv 18 Mar 2022.

Builds on

69 of 76 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2016Layer Normalizationcited by 5,579
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2016Wide Residual Networkscited by 3,057
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  13. 2017Decoupled Weight Decay Regularizationcited by 1,722
  14. 2021ViViT: A Video Vision Transformercited by 1,610
  15. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  16. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  17. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  18. 2014How transferable are features in deep neural networks?cited by 1,216
  19. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  20. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  21. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  22. 2016Deep Networks with Stochastic Depthcited by 973
  23. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  24. 2021Multiscale Vision Transformerscited by 896
  25. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  26. 2021Going deeper with Image Transformerscited by 759
  27. 2021Transformer in Transformercited by 632
  28. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  29. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  30. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  31. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  32. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  33. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  34. 2019Selective Kernel Networkscited by 401
  35. 2021Early Convolutions Help Transformers See Bettercited by 395
  36. 2021Pay Attention to MLPscited by 367
  37. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  38. 2019Attention Augmented Convolutional Networkscited by 363

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 23; 2024: 33; 2025: 12; 2026: 9.

90 papers.

Sort:

By area:

  1. 2022DeiT III: Revenge of the ViTcited by 417
  2. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  3. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  4. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  5. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  6. 2022Masked World Models for Visual Controlcited by 169
  7. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  8. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  9. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  10. 2023RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Drivingcited by 76
  11. 2022LPT: Long-tailed Prompt Tuning for Image Classificationcited by 69
  12. 2023A Billion-scale Foundation Model for Remote Sensing Imagescited by 62
38 more
  1. 2023No Representation Rules Them All in Category Discoverycited by 61
  2. 2023Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunitycited by 54
  3. 2024Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biologycited by 52
  4. 2024Vision-LSTM: xLSTM as Generic Vision Backbonecited by 51
  5. 2022Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognitioncited by 45
  6. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  7. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  8. 2024How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Modelcited by 32
  9. 2022Dual Vision Transformercited by 29
  10. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  11. 2025Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And Morecited by 28
  12. 2025Walrus: A Cross-Domain Foundation Model for Continuum Dynamicscited by 26
  13. 2023Improving Pixel-based MIM by Reducing Wasted Modeling Capabilitycited by 24
  14. 2022Neural Integral Equationscited by 23
  15. 2023FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysiscited by 23
  16. 2023DAT++: Spatially Dynamic Vision Transformer with Deformable Attentioncited by 21
  17. 2023ERM++: An Improved Baseline for Domain Generalizationcited by 18
  18. 2024Low-Rank Adaptation of Time Series Foundational Models for Out-of-Domain Modality Forecastingcited by 18
  19. 2024Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learningcited by 17
  20. 2023Mitigating Transformer Overconfidence via Lipschitz Regularizationcited by 15
  21. 2024DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasetscited by 14
  22. 2023Zorro: the masked multimodal transformercited by 13
  23. 2023Domain Generalization Using Large Pretrained Models with Mixture-of-Adapterscited by 11
  24. 2026Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptationcited by 11
  25. 2022Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammarscited by 9
  26. 2023CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformerscited by 9
  27. 2023Masked Autoencoders are Scalable Learners of Cellular Morphologycited by 9
  28. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 9
  29. 2024Is Large-Scale Pretraining the Secret to Good Domain Generalization?cited by 9
  30. 2025SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentationcited by 8
  31. 2023Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformerscited by 7
  32. 2024Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classificationcited by 7
  33. 2022ViT-DD: Multi-Task Vision Transformer for Semi-Supervised Driver Distraction Detectioncited by 6
  34. 2024Selective Visual Prompting in Vision Mambacited by 6
  35. 2024A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentationcited by 5
  36. 2024How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks?cited by 5
  37. 2025MHAFF: Multi-Head Attention Feature Fusion of CNN and Transformer for Cattle Identificationcited by 5
  38. 2024Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classificationcited by 4

50 most cited of 90 All 90 (JSON)