mlsanityData to 7 Oct 2026

Video Transformer Network

arXiv 2102.00719 ↗First posted 1 Feb 2021cs.CVcited by 271

Not on the map. Why?

Abstract (arXiv)

This paper presents VTN, a transformer-based framework for video recognition. Inspired by recent developments in vision transformers, we ditch the standard approach in video action recognition that relies on 3D ConvNets and introduce a method that classifies actions by attending to the entire video sequence information. Our approach is generic and builds on top of any given 2D spatial network. In terms of wall runtime, it trains 16.1× faster and runs 5.1× faster during inference while maintaining competitive accuracy compared to other state-of-the-art methods. It enables whole video analysis, via a single end-to-end pass, while requiring 1.5× fewer GFLOPs. We report competitive results on Kinetics-400 and present an ablation study of VTN properties and the trade-off between accuracy and inference speed. We hope our approach will serve as a new baseline and start a fresh line of research in the video recognition domain. Code and models are available at: https://github.com/bomri/SlowFast/blob/master/projects/vtn/README.md

Key facts. Cited by 271 papers in the map (data through 7 Oct 2026). Builds on 34 of its 42 references that are in the map. First posted on arXiv 1 Feb 2021.

Builds on

34 of 42 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2017Mask R-CNNcited by 7,259
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
22 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2015Fast R-CNNcited by 2,930
  4. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2018SlowFast Networks for Video Recognitioncited by 1,862
  10. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  11. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  12. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  13. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  14. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  15. 2018Video Action Transformer Networkcited by 316
  16. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  17. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  18. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  19. 2018Rethinking the Faster R-CNN Architecture for Temporal Action Localizationcited by 194
  20. 2016Spatiotemporal Residual Networks for Video Action Recognitioncited by 91
  21. 2019Only Time Can Tell: Discovering Temporal Data for Temporal Modelingcited by 66
  22. 2019AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architecturescited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 80; 2023: 70; 2024: 41; 2025: 19; 2026: 10.

271 papers.

Sort:

By area:

  1. 2021ViViT: A Video Vision Transformercited by 1,610
  2. 2021Video Swin Transformercited by 1,090
  3. 2021Multiscale Vision Transformerscited by 896
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2021ImageNet-21K Pretraining for the Massescited by 607
  6. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  7. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  8. 2022DeiT III: Revenge of the ViTcited by 417
  9. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  10. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  11. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  12. 2022VRT: A Video Restoration Transformercited by 219
38 more
  1. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  2. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  3. 2021Anticipative Video Transformercited by 197
  4. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  5. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  6. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  7. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  8. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  9. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
  10. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  11. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 146
  12. 2021PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformercited by 118
  13. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  14. 2021VidTr: Video Transformer Without Convolutionscited by 112
  15. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  16. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  17. 2023TALL: Thumbnail Layout for Deepfake Video Detectioncited by 100
  18. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  19. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 90
  20. 2021Long Short-Term Transformer for Online Action Detectioncited by 87
  21. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  22. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Datacited by 84
  23. 2021TVT: Transferable Vision Transformer for Unsupervised Domain Adaptationcited by 81
  24. 2021Is Attention Better Than Matrix Decomposition?cited by 75
  25. 2022VCT: A Video Compression Transformercited by 74
  26. 2021SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decompositioncited by 69
  27. 2022SVFormer: Semi-supervised Video Transformer for Action Recognitioncited by 67
  28. 2022Video Transformers: A Surveycited by 63
  29. 2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 59
  30. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 58
  31. 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
  32. 2023GETNext: Trajectory Flow Map Enhanced Transformer for Next POI Recommendationcited by 57
  33. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  34. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  35. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  36. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 55
  37. 2022Temporally Efficient Vision Transformer for Video Instance Segmentationcited by 55
  38. 2023ViTs for SITS: Vision Transformers for Satellite Image Time Seriescited by 55

50 most cited of 271 All 271 (JSON)