Data to 7 Oct 2026

A Survey on Visual Transformer

arXiv 2012.12556 ↗First posted 23 Dec 2020cs.CV cs.AIcited by 653

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer, first applied to the field of natural language processing, is a type of deep neural network mainly based on the self-attention mechanism. Thanks to its strong representation capabilities, researchers are looking at ways to apply transformer to computer vision tasks. In a variety of visual benchmarks, transformer-based models perform similar to or better than other types of networks such as convolutional and recurrent neural networks. Given its high performance and less need for vision-specific inductive bias, transformer is receiving more and more attention from the computer vision community. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages. The main categories we explore include the backbone network, high/mid-level vision, low-level vision, and video processing. We also include efficient transformer methods for pushing transformer into real device-based applications. Furthermore, we also take a brief look at the self-attention mechanism in computer vision, as it is the base component in transformer. Toward the end of this paper, we discuss the challenges and provide several further research directions for vision transformers.

Key facts. Cited by 653 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 81; 2023: 178; 2024: 165; 2025: 151; 2026: 70.

653 papers.

Sort:

By area:

2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
2021Transformer in Transformercited by 632
2022Transformers in Time Series: A Surveycited by 489
2022Multimodal Learning with Transformers: A Surveycited by 399
2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
2022Vision GNN: An Image is Worth Graph of Nodescited by 185
2024Swin-UMamba: Mamba-based UNet with ImageNet-based pretrainingcited by 179
2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
2023Optimal Transport Aggregation for Visual Place Recognitioncited by 151
2023Spike-driven Transformercited by 141

38 more

2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
2023Transformer-Based Visual Segmentation: A Surveycited by 124
2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
2021A Survey of Visual Transformerscited by 117
2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 111
2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
2022An Empirical Study of Remote Sensing Pretrainingcited by 105
2022Attention Spiking Neural Networkscited by 103
2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
2022MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detectioncited by 87
2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
2023Attending to Graph Transformerscited by 81
2022P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimationcited by 80
2022Unified Visual Transformer Compressioncited by 79
2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
2022E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Contextcited by 76
2023Semantic Segmentation using Vision Transformers: A surveycited by 73
2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
2023Beyond One-to-One: Rethinking the Referring Image Segmentationcited by 68
2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
2022Video Transformers: A Surveycited by 63
2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
2023AD-CLIP: Adapting Domains in Prompt Space Using CLIPcited by 62
2023On the Connection Between MPNN and Graph Transformercited by 60
2022Black-box Dataset Ownership Verification via Backdoor Watermarkingcited by 57
2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
2022A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localizationcited by 55
2022Multiple Instance Learning for Digital Pathology: A Review on the State-of-the-Art, Limitations & Future Potentialcited by 54
2023Source-Free Domain Adaptation with Frozen Multimodal Foundation Modelcited by 54
2024Hibou: A Family of Foundational Vision Transformers for Pathologycited by 52
2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
2023Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanismcited by 51
2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50

50 most cited of 653 All 653 (JSON)