mlsanityData to 14 Oct 2026

ViViT: A Video Vision Transformer

arXiv 2103.15691 ↗First posted 29 Mar 2021cs.CVcited by 1,620

Not on the map. Why?

Cited by 42 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatio-temporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks. To facilitate further research, we release code at https://github.com/google-research/scenic/tree/main/scenic/projects/vivit

Key facts. Cited by 1,620 papers in the map (data through 14 Oct 2026). Builds on 75 of its 87 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

75 of 87 references.

References · 75

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2016Layer Normalizationcited by 5,669
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2017Squeeze-and-Excitation Networkscited by 4,789
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
38 more
  1. 2014Going Deeper with Convolutionscited by 4,241
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2017Non-local Neural Networkscited by 2,293
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  8. 2017The Kinetics Human Action Video Datasetcited by 1,909
  9. 2018SlowFast Networks for Video Recognitioncited by 1,867
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  12. 2020Rethinking Attention with Performerscited by 1,601
  13. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  14. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  15. 2020Point Transformercited by 1,470
  16. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  17. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  18. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  19. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  20. 2016Deep Networks with Stochastic Depthcited by 979
  21. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  22. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  23. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  24. 2020Efficient Transformers: A Surveycited by 790
  25. 2018Image Transformercited by 772
  26. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 718
  27. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  28. 2018Universal Transformerscited by 647
  29. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  30. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  31. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  32. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  33. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  34. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  35. 2019Attention Augmented Convolutional Networkscited by 363
  36. 2019Axial Attention in Multidimensional Transformerscited by 358
  37. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  38. 2018A Short Note about Kinetics-600cited by 333

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 110; 2022: 333; 2023: 362; 2024: 327; 2025: 291; 2026: 197.

1,620 papers.

Sort:

By area:

  1. 2022Video Diffusion Modelscited by 2,305
  2. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  5. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  6. 2021Video Swin Transformercited by 1,091
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  8. 2021Multiscale Vision Transformerscited by 897
  9. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  11. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  12. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
38 more
  1. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  3. 2023ModelScope Text-to-Video Technical Reportcited by 658
  4. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  6. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  7. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  8. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  9. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  10. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  11. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  12. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  13. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  14. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  15. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  16. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  17. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  18. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  19. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  20. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  21. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 408
  22. 2021A Survey of Transformerscited by 401
  23. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  24. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  25. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  26. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  27. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  28. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  29. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 311
  30. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  31. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  32. 2022MAXIM: Multi-Axis MLP for Image Processingcited by 302
  33. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
  34. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 280
  35. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  36. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  37. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  38. 2023A Cookbook of Self-Supervised Learningcited by 250

50 most cited of 1,620 All 1,620 (JSON)