mlsanityData to 7 Oct 2026

Multiview Transformers for Video Recognition

arXiv 2201.04288 ↗First posted 12 Jan 2022cs.CV cs.LGcited by 148

Not on the map. Why?

Abstract (arXiv)

Video understanding requires reasoning at multiple spatiotemporal resolutions -- from short fine-grained motions to events taking place over longer durations. Although transformer architectures have recently advanced the state-of-the-art, they have not explicitly modelled different spatiotemporal resolutions. To this end, we present Multiview Transformers for Video Recognition (MTV). Our model consists of separate encoders to represent different views of the input video with lateral connections to fuse information across views. We present thorough ablation studies of our model and show that MTV consistently performs better than single-view counterparts in terms of accuracy and computational cost across a range of model sizes. Furthermore, we achieve state-of-the-art results on six standard datasets, and improve even further with large-scale pretraining. Code and checkpoints are available at: https://github.com/google-research/scenic/tree/main/scenic/projects/mtv.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 68 of its 85 references that are in the map. First posted on arXiv 12 Jan 2022.

Builds on

68 of 85 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2016Layer Normalizationcited by 5,579
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
38 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2014Going Deeper with Convolutionscited by 4,231
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  7. 2016Pyramid Scene Parsing Networkcited by 2,472
  8. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2017The Kinetics Human Action Video Datasetcited by 1,894
  13. 2018SlowFast Networks for Video Recognitioncited by 1,862
  14. 2021ViViT: A Video Vision Transformercited by 1,610
  15. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  16. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  17. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  18. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  19. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  20. 2021Scaling Vision Transformerscited by 993
  21. 2016Deep Networks with Stochastic Depthcited by 973
  22. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  23. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  24. 2021Multiscale Vision Transformerscited by 896
  25. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  26. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  27. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  28. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  29. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  30. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  31. 2020Rescaling Egocentric Visioncited by 529
  32. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  33. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  34. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  35. 2017Large-Scale Evolution of Image Classifierscited by 348
  36. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  37. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  38. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 50; 2023: 46; 2024: 27; 2025: 15; 2026: 10.

148 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  3. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  4. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  5. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  6. 2022Multimodal Learning with Transformers: A Surveycited by 399
  7. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  8. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  9. 2022Learning Video Representations from Large Language Modelscited by 204
  10. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  11. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  12. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
38 more
  1. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  2. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 146
  3. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  4. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  5. 2022HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutionscited by 103
  6. 2023PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimationcited by 91
  7. 2023An Outlook into the Future of Egocentric Visioncited by 84
  8. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  9. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  10. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 70
  11. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  12. 2022Video Transformers: A Surveycited by 63
  13. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
  14. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 48
  15. 2022Audiovisual Masked Autoencoderscited by 47
  16. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  17. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 44
  18. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 41
  19. 2023Dual-path Adaptation from Image to Video Transformerscited by 40
  20. 2022Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videoscited by 39
  21. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  22. 2023Transformers in Speech Processing: A Surveycited by 38
  23. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  24. 2023Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learningcited by 31
  25. 2023Ego-Only: Egocentric Action Detection without Exocentric Transferringcited by 30
  26. 2023VicTR: Video-conditioned Text Representations for Activity Recognitioncited by 28
  27. 2022Vision Transformers for Action Recognition: A Surveycited by 25
  28. 2022Recent Advances in Vision Transformer: A Survey and Outlook of Recent Workcited by 21
  29. 2023Incomplete Multi-view Clustering via Prototype-based Imputationcited by 21
  30. 2023How can objects help action recognition?cited by 21
  31. 2023Training a Large Video Model on a Single Machine in a Daycited by 20
  32. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  33. 2022M&M Mix: A Multimodal Multiview Transformer Ensemblecited by 19
  34. 2022SCOTCH and SODA: A Transformer Video Shadow Detection Frameworkcited by 19
  35. 2022Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localizationcited by 19
  36. 2023From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understandingcited by 19
  37. 2023Video-FocalNets: Spatio-Temporal Focal Modulation for Video Action Recognitioncited by 18
  38. 2023Human-centric Scene Understanding for 3D Large-scale Scenarioscited by 17

50 most cited of 148 All 148 (JSON)