mlsanityData to 7 Oct 2026

Vision Transformers: State of the Art and Research Challenges

arXiv 2207.03041 ↗First posted 7 Jul 2022cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Transformers have achieved great success in natural language processing. Due to the powerful capability of self-attention mechanism in transformers, researchers develop the vision transformers for a variety of computer vision tasks, such as image recognition, object detection, image segmentation, pose estimation, and 3D reconstruction. This paper presents a comprehensive overview of the literature on different architecture designs and training tricks (including self-supervised learning) for vision transformers. Our goal is to provide a systematic review with the open research opportunities.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 48 of its 57 references that are in the map. First posted on arXiv 7 Jul 2022.

Builds on

48 of 57 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
36 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  12. 2016Deep Networks with Stochastic Depthcited by 973
  13. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  14. 2020On Layer Normalization in the Transformer Architecturecited by 906
  15. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  16. 2021Going deeper with Image Transformerscited by 759
  17. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  18. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  19. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  20. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  21. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  22. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  23. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  24. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  25. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  26. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  27. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  28. 2019Fixing the train-test resolution discrepancycited by 209
  29. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  30. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  31. 2021Towards Robust Vision Transformercited by 137
  32. 2021Self-Supervised Learning with Swin Transformerscited by 121
  33. 2021SiT: Self-supervised vIsion Transformercited by 99
  34. 2021Medical Image Segmentation Using Squeeze-and-Expansion Transformerscited by 56
  35. 2021Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Textcited by 14
  36. 2021Laughing Heads: Can Transformers Detect What Makes a Sentence Funny?cited by 9

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 3.

7 papers.

  1. 2022Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learningcited by 78
  2. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  3. 2025Enhancing Transformer-Based Vision Models: Addressing Feature Map Anomalies Through Novel Optimization Strategiescited by 2
  4. 2024Performance of computer vision algorithms for fine-grained classification using crowdsourced insect imagescited by 1
  5. 2024DST-GTN: Dynamic Spatio-Temporal Graph Transformer Network for Traffic Forecastingcited by 1
  6. 2023A Novel Site-Agnostic Multimodal Deep Learning Model to Identify Pro-Eating Disorder Content on Social Mediacited by 0
  7. 2025ALERT Open Dataset and Input-Size-Agnostic Vision Transformer for Driver Activity Recognition using IR-UWBcited by 0