mlsanityData to 14 Oct 2026

Expanding Language-Image Pretrained Models for General Video Recognition

arXiv 2208.02816 ↗First posted 4 Aug 2022cs.CVcited by 321

Not on the map. Why?

Cited by 48 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive language-image pretraining has shown great success in learning visual-textual joint representation from web-scale data, demonstrating remarkable "zero-shot" generalization ability for various image tasks. However, how to effectively expand such new language-image pretraining methods to video domains is still an open problem. In this work, we present a simple yet effective approach that adapts the pretrained language-image models to video recognition directly, instead of pretraining a new model from scratch. More concretely, to capture the long-range dependencies of frames along the temporal dimension, we propose a cross-frame attention mechanism that explicitly exchanges information across frames. Such module is lightweight and can be plugged into pretrained language-image models seamlessly. Moreover, we propose a video-specific prompting scheme, which leverages video content information for generating discriminative textual prompts. Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited. Code and models are available at https://aka.ms/X-CLIP

Key facts. Cited by 321 papers in the map (data through 14 Oct 2026). Builds on 51 of its 64 references that are in the map. First posted on arXiv 4 Aug 2022.

Builds on

51 of 64 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2016Layer Normalizationcited by 5,669
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  8. 2017The Kinetics Human Action Video Datasetcited by 1,909
  9. 2018SlowFast Networks for Video Recognitioncited by 1,867
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  11. 2021ViViT: A Video Vision Transformercited by 1,620
  12. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
38 more
  1. 2021Video Swin Transformercited by 1,091
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  4. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  5. 2021Multiscale Vision Transformerscited by 897
  6. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  8. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  9. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  10. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  11. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  12. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  13. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  14. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  15. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 387
  16. 2018A Short Note about Kinetics-600cited by 333
  17. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  18. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  19. 2021Video Transformer Networkcited by 272
  20. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  21. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  22. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  23. 2021Anticipative Video Transformercited by 200
  24. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  25. 2015Label-Embedding for Image Classificationcited by 165
  26. 2020TEA: Temporal Excitation and Aggregation for Action Recognitioncited by 165
  27. 2016Learning a Deep Embedding Model for Zero-Shot Learningcited by 153
  28. 2022Multiview Transformers for Video Recognitioncited by 148
  29. 2020TAM: Temporal Adaptive Module for Video Recognitioncited by 139
  30. 2020A Comprehensive Study of Deep Video Action Recognitioncited by 88
  31. 2020Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applicationscited by 80
  32. 2021Elaborative Rehearsal for Zero-shot Action Recognitioncited by 77
  33. 2022Video Transformers: A Surveycited by 63
  34. 2016Going Deeper into Action Recognition: A Surveycited by 55
  35. 2018Towards Universal Representation for Unseen Action Recognitioncited by 44
  36. 2021Co-training Transformer with Videos and Images Improves Action Recognitioncited by 41
  37. 2016Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentationcited by 27
  38. 2017Alternative Semantic Representations for Zero-Shot Human Action Recognitioncited by 27

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 17; 2023: 87; 2024: 102; 2025: 66; 2026: 49.

321 papers.

Sort:

By area:

  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  2. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  3. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  4. 2023A Cookbook of Self-Supervised Learningcited by 250
  5. 2022Learning Video Representations from Large Language Modelscited by 205
  6. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  7. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  8. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  9. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  10. 2023VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detectioncited by 123
  11. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  12. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
38 more
  1. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  2. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 116
  3. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  4. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  5. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 92
  6. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 91
  7. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  8. 2022CLIP-Driven Fine-grained Text-Image Person Re-identificationcited by 79
  9. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  10. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  11. 2023Diffusion Models as Masked Autoencoderscited by 71
  12. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 70
  13. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 66
  14. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  15. 2023Unbiased Multiple Instance Learning for Weakly Supervised Video Anomaly Detectioncited by 64
  16. 20244Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Modelscited by 64
  17. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 61
  18. 2023Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferringcited by 58
  19. 2023Diversity-Aware Meta Visual Promptingcited by 58
  20. 2023Open-Vocabulary Video Anomaly Detectioncited by 57
  21. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  22. 2023Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimizationcited by 55
  23. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Modelcited by 52
  24. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  25. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 51
  26. 2024Towards A Better Metric for Text-to-Video Generationcited by 51
  27. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  28. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
  29. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 49
  30. 2023Breathing Life Into Sketches Using Text-to-Video Priorscited by 48
  31. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 46
  32. 2023Towards Explainable In-the-Wild Video Quality Assessment: A Database and a Language-Prompted Approachcited by 45
  33. 2023Prompting Visual-Language Models for Dynamic Facial Expression Recognitioncited by 44
  34. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
  35. 2024Abductive Ego-View Accident Video Understanding for Safe Driving Perceptioncited by 43
  36. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  37. 2023CLIP-guided Prototype Modulating for Few-shot Action Recognitioncited by 41
  38. 2023Dual-path Adaptation from Image to Video Transformerscited by 41

50 most cited of 321 All 321 (JSON)