# Recent Advances in Vision Transformer: A Survey and Outlook of Recent Work

arXiv: https://arxiv.org/abs/2203.01536 · HTML: https://mlsanity.com/paper/2203.01536/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 61 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [LocalViT: Analyzing Locality in Vision Transformers](https://mlsanity.com/paper/2104.05707/) (2021)
- [VRT: A Video Restoration Transformer](https://mlsanity.com/paper/2201.12288/) (2022)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [3D Object Detection with Pointformer](https://mlsanity.com/paper/2012.11409/) (2020)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [VOLO: Vision Outlooker for Visual Recognition](https://mlsanity.com/paper/2106.13112/) (2021)
- [StyleSwin: Transformer-based GAN for High-resolution Image Generation](https://mlsanity.com/paper/2112.10762/) (2021)
- [Multiview Transformers for Video Recognition](https://mlsanity.com/paper/2201.04288/) (2022)
- [TransFG: A Transformer Architecture for Fine-grained Recognition](https://mlsanity.com/paper/2103.07976/) (2021)
- [Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction](https://mlsanity.com/paper/2103.12091/) (2021)

## Built on by (most cited first, 21 of 21)

- [E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning](https://mlsanity.com/paper/2307.13770/) (2023)
- [Rethinking Mobile Block for Efficient Attention-based Models](https://mlsanity.com/paper/2301.01146/) (2023)
- [FT2TF: First-Person Statement Text-To-Talking Face Generation](https://mlsanity.com/paper/2312.05430/) (2023)
- [A Recent Survey of Vision Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2312.00634/) (2023)
- [GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editing](https://mlsanity.com/paper/2412.02366/) (2024)
- [Occluded Person Re-Identification with Deep Learning: A Survey and Perspectives](https://mlsanity.com/paper/2311.00603/) (2023)
- [AMD: Automatic Multi-step Distillation of Large-scale Vision Models](https://mlsanity.com/paper/2407.04208/) (2024)
- [Transformers in Single Object Tracking: An Experimental Survey](https://mlsanity.com/paper/2302.11867/) (2023)
- [Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers](https://mlsanity.com/paper/2308.09372/) (2023)
- [SAIC: Integration of Speech Anonymization and Identity Classification](https://mlsanity.com/paper/2312.15190/) (2023)
- [EMOv2: Pushing 5M Vision Model Frontier](https://mlsanity.com/paper/2412.06674/) (2024)
- [Generative Pre-training for Subjective Tasks: A Diffusion Transformer-Based Framework for Facial Beauty Prediction](https://mlsanity.com/paper/2507.20363/) (2025)
- [Foundation Models for Autonomous Robots in Unstructured Environments](https://mlsanity.com/paper/2407.14296/) (2024)
- [SkyGPT: Probabilistic Short-term Solar Forecasting Using Synthetic Sky Videos from Physics-constrained VideoGPT](https://mlsanity.com/paper/2306.11682/) (2023)
- [OCT-SelfNet: A Self-Supervised Framework with Multi-Modal Datasets for Generalized and Robust Retinal Disease Detection](https://mlsanity.com/paper/2401.12344/) (2024)
- [Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification](https://mlsanity.com/paper/2409.11375/) (2024)
- [Sequence and Circle: Exploring the Relationship Between Patches](https://mlsanity.com/paper/2210.09871/) (2022)
- [RevealED: Uncovering Pro-Eating Disorder Content on Twitter Using Deep Learning](https://mlsanity.com/paper/2212.13949/) (2022)
- [Towards In-Vehicle Multi-Task Facial Attribute Recognition: Investigating Synthetic Data and Vision Foundation Models](https://mlsanity.com/paper/2403.06088/) (2024)
- [Towards Foundation Models for Cryo-ET Subtomogram Analysis](https://mlsanity.com/paper/2509.24311/) (2025)
- [Hierarchical Reinforcement Learning for Neural Network Compression (HiReLC): Pruning and Quantization](https://mlsanity.com/paper/2606.26002/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs) are becoming more popular and dominating technique for various vision tasks, compare to Convolutional Neural Networks (CNNs). As a demanding technique in computer vision, ViTs have been successfully solved various vision problems while focusing on long-range relationships. In this paper, we begin by introducing the fundamental concepts and background of the self-attention mechanism. Next, we provide a comprehensive overview of recent top-performing ViT methods describing in terms of strength and weakness, computational cost as well as training and testing dataset. We thoroughly compare the performance of various ViT algorithms and most representative CNN methods on popular benchmark datasets. Finally, we explore some limitations with insightful observations and provide further research direction. The project page along with the collections of papers are available at https://github.com/khawar512/ViT-Survey