mlsanityData to 7 Oct 2026

Transformers in Vision: A Survey

arXiv 2101.01169 ↗First posted 4 Jan 2021cs.CV cs.AI cs.LGcited by 1,009

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities (e.g., images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional encoding. We then cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering, visual reasoning, and visual grounding), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution, image enhancement, and colorization) and 3D analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works.

Key facts. Cited by 1,009 papers in the map (data through 7 Oct 2026). Builds on 167 of its 286 references that are in the map. First posted on arXiv 4 Jan 2021.

Builds on

167 of 286 references.

References · 167

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  11. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  12. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
38 more
  1. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2017Mask R-CNNcited by 7,259
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2016Layer Normalizationcited by 5,579
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  13. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  14. 2013Auto-Encoding Variational Bayescited by 3,980
  15. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  16. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  17. 2020Longformer: The Long-Document Transformercited by 3,362
  18. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  19. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  20. 2015Fast R-CNNcited by 2,930
  21. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  22. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  23. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  24. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  25. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  26. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  27. 2015VQA: Visual Question Answeringcited by 2,489
  28. 2017Non-local Neural Networkscited by 2,290
  29. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  30. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  31. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  32. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  33. 2017The Kinetics Human Action Video Datasetcited by 1,894
  34. 2020Reformer: The Efficient Transformercited by 1,658
  35. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  36. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  37. 2020Rethinking Attention with Performerscited by 1,584
  38. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548

50 most cited of 167 All 167 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 148; 2022: 187; 2023: 245; 2024: 181; 2025: 173; 2026: 75.

1,009 papers.

Sort:

By area:

  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  2. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  3. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  4. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  5. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  7. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  8. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  9. 2022Transformers in Time Series: A Surveycited by 489
  10. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  11. 2021Intriguing Properties of Vision Transformerscited by 446
  12. 2021TransReID: Transformer-based Object Re-Identificationcited by 432
38 more
  1. 2021Dynamic Neural Networks: A Surveycited by 405
  2. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  3. 2021A Survey of Transformerscited by 399
  4. 2022Multimodal Learning with Transformers: A Surveycited by 399
  5. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  6. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  7. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 271
  8. 2021Mesh Graphormercited by 264
  9. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  10. 2021Curriculum Learning: A Surveycited by 257
  11. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  12. 2022Transformers in Medical Imaging: A Surveycited by 245
  13. 2022A Transformer-Based Siamese Network for Change Detectioncited by 243
  14. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  15. 2023Looped Transformers as Programmable Computerscited by 224
  16. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  17. 2021Deep Contextual Video Compressioncited by 216
  18. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  19. 2023I2SB: Image-to-Image Schr"odinger Bridgecited by 214
  20. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  21. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 201
  22. 2022FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operatorscited by 188
  23. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  24. 2022Focal Modulation Networkscited by 186
  25. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  26. 2022Translation between Molecules and Natural Languagecited by 173
  27. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  28. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  29. 2022Learning Enriched Features for Fast Image Restoration and Enhancementcited by 150
  30. 2023Spike-driven Transformercited by 141
  31. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  32. 2022On The Computational Complexity of Self-Attentioncited by 137
  33. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  34. 2022Deep Learning for Time Series Anomaly Detection: A Surveycited by 130
  35. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  36. 2021OW-DETR: Open-world Detection Transformercited by 129
  37. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  38. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127

50 most cited of 1,009 All 1,009 (JSON)