mlsanityData to 7 Oct 2026

Video Transformers: A Survey

arXiv 2201.05991 ↗First posted 16 Jan 2022cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

Transformer models have shown great success handling long-range interactions, making them a promising tool for modeling video. However, they lack inductive biases and scale quadratically with input length. These limitations are further exacerbated when dealing with the high dimensionality introduced by the temporal dimension. While there are surveys analyzing the advances of Transformers for vision, none focus on an in-depth analysis of video-specific designs. In this survey, we analyze the main contributions and trends of works leveraging Transformers to model video. Specifically, we delve into how videos are handled at the input level first. Then, we study the architectural changes made to deal with video more efficiently, reduce redundancy, re-introduce useful inductive biases, and capture long-term temporal dynamics. In addition, we provide an overview of different training regimes and explore effective self-supervised learning strategies for video. Finally, we conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D ConvNets even with less computational complexity.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 219 of its 288 references that are in the map. First posted on arXiv 16 Jan 2022.

Builds on

219 of 288 references.

References · 219

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2017Graph Attention Networkscited by 8,854
  12. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
38 more
  1. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2017Mask R-CNNcited by 7,259
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2016Layer Normalizationcited by 5,579
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2017Squeeze-and-Excitation Networkscited by 4,763
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2017Neural Discrete Representation Learningcited by 4,650
  13. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  14. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  15. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  16. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  17. 2014Going Deeper with Convolutionscited by 4,231
  18. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  19. 2020Longformer: The Long-Document Transformercited by 3,362
  20. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  21. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  22. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  23. 2018YOLOv3: An Incremental Improvementcited by 2,652
  24. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  25. 2017Non-local Neural Networkscited by 2,290
  26. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  27. 2019Searching for MobileNetV3cited by 1,984
  28. 2018SlowFast Networks for Video Recognitioncited by 1,862
  29. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  30. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  31. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  32. 2020Reformer: The Efficient Transformercited by 1,658
  33. 2021ViViT: A Video Vision Transformercited by 1,610
  34. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  35. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  36. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  37. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  38. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424

50 most cited of 219 All 219 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 21; 2024: 15; 2025: 10; 2026: 7.

63 papers.

Sort:

By area:

  1. 2022Transformers in Time Series: A Surveycited by 489
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  4. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  5. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  6. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  7. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  8. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  9. 2022Transformers in Remote Sensing: A Surveycited by 61
  10. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  11. 20223D Vision with Transformers: A Surveycited by 21
  12. 2023COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformerscited by 17
38 more
  1. 2024On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Eracited by 17
  2. 2023Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semi-Supervised Skeleton-based Action Recognitioncited by 16
  3. 2023DiVa-360: The Dynamic Visual Dataset for Immersive Neural Fieldscited by 16
  4. 2023Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learningcited by 16
  5. 2023The Unreasonable Effectiveness of Large Language-Vision Models for Source-free Video Domain Adaptationcited by 16
  6. 2023An Overview on Generative AI at Scale with Edge-Cloud Computingcited by 9
  7. 2023Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformerscited by 9
  8. 2023DISeR: Designing Imaging Systems with Reinforcement Learningcited by 6
  9. 2025Can masking background and object reduce static bias for zero-shot action recognition?cited by 6
  10. 2023Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extractioncited by 5
  11. 2022Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attentioncited by 4
  12. 2023Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videoscited by 4
  13. 2024Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insightscited by 4
  14. 2022Performance Evaluation of Action Recognition Models on Low Quality Videoscited by 3
  15. 2022Less is More: Facial Landmarks can Recognize a Spontaneous Smilecited by 3
  16. 2023StreamingFlow: Streaming Occupancy Forecasting with Asynchronous Multi-modal Data Streams via Neural Ordinary Differential Equationcited by 3
  17. 2024OCK: Unsupervised Dynamic Video Prediction with Object-Centric Kinematicscited by 3
  18. 2024SVFormer: A Direct Training Spiking Transformer for Efficient Video Action Recognitioncited by 3
  19. 2024SurgPETL: Parameter-Efficient Image-to-Surgical-Video Transfer Learning for Surgical Phase Recognitioncited by 3
  20. 2025Context Guided Transformer Entropy Modeling for Video Compressioncited by 3
  21. 2024ESP-PCT: Enhanced VR Semantic Performance through Efficient Compression of Temporal and Spatial Redundancies in Point Cloud Transformerscited by 2
  22. 2024Scaling Law Hypothesis for Multimodal Modelcited by 2
  23. 2024Domain Aware Multi-Task Pretraining of 3D Swin Transformer for T1-weighted Brain MRIcited by 2
  24. 2024STAA: Spatio-Temporal Attention Attribution for Real-Time Interpreting Transformer-based Video Modelscited by 2
  25. 2024Reversing the Damage: A QP-Aware Transformer-Diffusion Approach for 8K Video Restoration under Codec Compressioncited by 2
  26. 2026VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluationcited by 2
  27. 2022Ego Vehicle Speed Estimation using 3D Convolution with Masked Attentioncited by 1
  28. 2024No Time to Waste: Squeeze Time into Channel for Mobile Video Understandingcited by 1
  29. 2024Artificial Inductive Bias for Synthetic Tabular Data Generation in Data-Scarce Scenarioscited by 1
  30. 2024Wavelet GPT: Wavelet Inspired Large Language Modelscited by 1
  31. 2025Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generationcited by 1
  32. 2025Separating Shared and Domain-Specific LoRAs for Multi-Domain Learningcited by 1
  33. 2025Prompting with Sign Parameters for Low-resource Sign Language Instruction Generationcited by 1
  34. 2026KiToke: Kernel-based Interval-aware Token Compression for Video Large Language Modelscited by 1
  35. 2022Object-ABN: Learning to Generate Sharp Attention Maps for Action Recognitioncited by 0
  36. 2023Seeing the Pose in the Pixels: Learning Pose-Aware Representations in Vision Transformerscited by 0
  37. 2023Group Activity Recognition in Computer Vision: A Comprehensive Review, Challenges, and Future Perspectivescited by 0
  38. 2023Weakly Supervised AI for Efficient Analysis of 3D Pathology Samplescited by 0

50 most cited of 63 All 63 (JSON)