mlsanityData to 7 Oct 2026

Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning

arXiv 2212.03229 ↗First posted 6 Dec 2022cs.CVcited by 49

Not on the map. Why?

Abstract (arXiv)

We present a simple approach which can turn a ViT encoder into an efficient video model, which can seamlessly work with both image and video inputs. By sparsely sampling the inputs, the model is able to do training and inference from both inputs. The model is easily scalable and can be adapted to large-scale pre-trained ViTs without requiring full finetuning. The model achieves SOTA results and the code will be open-sourced.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 68 of its 76 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

68 of 76 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2018SlowFast Networks for Video Recognitioncited by 1,862
  12. 2021ViViT: A Video Vision Transformercited by 1,610
38 more
  1. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  6. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  7. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  8. 2021Video Swin Transformercited by 1,090
  9. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  10. 2021Scaling Vision Transformerscited by 993
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2021Multiscale Vision Transformerscited by 896
  13. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  14. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  15. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  16. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  17. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  18. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  19. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  20. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  21. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  22. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  23. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  24. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  25. 2021Intriguing Properties of Vision Transformerscited by 446
  26. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  27. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  28. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  29. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  30. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  31. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  32. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  33. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  34. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  35. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  36. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  37. 2022Multiview Transformers for Video Recognitioncited by 148
  38. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 16; 2025: 14; 2026: 4.

49 papers.

Sort:

By area:

  1. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  2. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  3. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 116
  4. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  5. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  6. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  7. 2023MaMMUT: A Simple Architecture for Joint Learning for MultiModal Taskscited by 24
  8. 2023Human-centric Scene Understanding for 3D Large-scale Scenarioscited by 17
  9. 2023Adapting Short-Term Transformers for Action Detection in Untrimmed Videoscited by 16
  10. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  11. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  12. 2025EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activitycited by 15
37 more
  1. 2023ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoderscited by 14
  2. 2023VideoGLUE: Video General Understanding Evaluation of Foundation Modelscited by 14
  3. 2024Towards a vision foundation model for comprehensive assessment of Cardiac MRIcited by 13
  4. 2025OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learningcited by 13
  5. 2023Flow Dynamics Correction for Action Recognitioncited by 12
  6. 2023Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Trainingcited by 12
  7. 2023COSA: Concatenated Sample Pretrained Vision-Language Foundation Modelcited by 11
  8. 2023Learning from One Continuous Video Streamcited by 11
  9. 2024Computer Vision for Primate Behavior Analysis in the Wildcited by 10
  10. 2024Snakes and Ladders: Two Steps Up for VideoMambacited by 10
  11. 2024Cross-Modal Learning for Anomaly Detection in Complex Industrial Process: Methodology and Benchmarkcited by 8
  12. 2024C2C: Component-to-Composition Learning for Zero-Shot Compositional Action Recognitioncited by 8
  13. 2024Enhancing Video Transformers for Action Understanding with VLM-aided Trainingcited by 6
  14. 2024Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graphcited by 6
  15. 2024TRecViT: A Recurrent Video Transformercited by 5
  16. 2025D'ej`a Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reusecited by 5
  17. 2023Optimizing ViViT Training: Time and Memory Reduction for Action Recognitioncited by 4
  18. 2024A Survey of IMU Based Cross-Modal Transfer Learning in Human Activity Recognitioncited by 4
  19. 2023CHAIN: Exploring Global-Local Spatio-Temporal Information for Improved Self-Supervised Video Hashingcited by 3
  20. 2024CardioSyntax: end-to-end SYNTAX score prediction -- dataset, benchmark and methodcited by 3
  21. 2024TAMT: Temporal-Aware Model Tuning for Cross-Domain Few-Shot Action Recognitioncited by 3
  22. 2025Feature Hallucination for Self-supervised Action Recognitioncited by 3
  23. 2024ParCon: Noise-Robust Collaborative Perception via Multi-module Parallel Connectioncited by 2
  24. 2025Self-supervised Normality Learning and Divergence Vector-guided Model Merging for Zero-shot Congenital Heart Disease Detection in Fetal Ultrasound Videoscited by 2
  25. 2025DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classificationcited by 2
  26. 2025Enhancing Video Large Language Models with Structured Multi-Video Collaborative Reasoningcited by 2
  27. 2025Text-driven Online Action Detectioncited by 1
  28. 2025Video CLIP Model for Multi-View Echocardiography Interpretationcited by 1
  29. 2025Cattle-CLIP: A Multimodal Framework for Cattle Behaviour Recognition from Videocited by 1
  30. 2025A Lightweight 3D-CNN for Event-Based Human Action Recognition with Privacy-Preserving Potentialcited by 1
  31. 2025CA3D: Convolutional-Attentional 3D Nets for Efficient Video Activity Recognition on the Edgecited by 0
  32. 2025Time-Scaling State-Space Models for Dense Video Captioningcited by 0
  33. 2025OmniFD: A Unified Model for Versatile Face Forgery Detectioncited by 0
  34. 2026SBF: An Effective Representation to Augment Skeleton for Video-based Human Action Recognitioncited by 0
  35. 2026Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformercited by 0
  36. 2026EAST: Early Action Prediction Sampling Strategy with Token Maskingcited by 0
  37. 2026Skeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretrainingcited by 0