mlsanityData to 7 Oct 2026

OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning

arXiv 2408.06158 ↗First posted 12 Aug 2024cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Recent Vision-Language Models (VLMs) textite.g. CLIP have made great progress in video recognition. Despite the improvement brought by the strong visual backbone in extracting spatial features, CLIP still falls short in capturing and integrating spatial-temporal features which is essential for video recognition. In this paper, we propose OmniCLIP, a framework that adapts CLIP for video recognition by focusing on learning comprehensive features encompassing spatial, temporal, and dynamic spatial-temporal scales, which we refer to as omni-scale features. This is achieved through the design of spatial-temporal blocks that include parallel temporal adapters (PTA), enabling efficient temporal modeling. Additionally, we introduce a self-prompt generator (SPG) module to capture dynamic object spatial features. The synergy between PTA and SPG allows OmniCLIP to discern varying spatial information across frames and assess object scales over time. We have conducted extensive experiments in supervised video recognition, few-shot video recognition, and zero-shot recognition tasks. The results demonstrate the effectiveness of our method, especially with OmniCLIP achieving a top-1 accuracy of 74.30% on HMDB51 in a 16-shot setting, surpassing the recent MotionPrompt approach even with full training data. The code is available at urlhttps://github.com/XiaoBuL/OmniCLIP.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 46 references that are in the map. First posted on arXiv 12 Aug 2024.

Builds on

39 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition22 Jan
  2. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models31 Dec

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  11. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
25 more
  1. 2021Video Swin Transformercited by 1,090
  2. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  3. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  4. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  5. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  6. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  7. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  8. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  9. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  10. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  11. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  12. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  13. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  14. 2016Spatiotemporal Residual Networks for Video Action Recognitioncited by 91
  15. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 90
  16. 2020Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applicationscited by 79
  17. 2021Elaborative Rehearsal for Zero-shot Action Recognitioncited by 76
  18. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 70
  19. 2023Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimizationcited by 54
  20. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 44
  21. 2024MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesiscited by 44
  22. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Modelcited by 41
  23. 2023Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learningcited by 16
  24. 2023Iterative Few-shot Semantic Segmentation from Image Label Textcited by 10
  25. 2023SYNC-CLIP: Synthetic Data Make CLIP Generalize Better in Data-Limited Scenarioscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  2. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  3. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  4. 2024TopoFR: A Closer Look at Topology Alignment on Face Recognitioncited by 13
  5. 2025On the Limitations of Vision-Language Models in Understanding Image Transformscited by 12
  6. 2024Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognitioncited by 10
  7. 2024Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmarkcited by 9
  8. 2025VEU-Bench: Towards Comprehensive Understanding of Video Editingcited by 7
  9. 2025Order Matters: On Parameter-Efficient Image-to-Video Probing for Recognizing Nearly Symmetric Actionscited by 3
  10. 2025ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignmentcited by 3
  11. 2026Frame2Freq: Spectral Adapters for Fine-Grained Video Understandingcited by 2
  12. 2025SPKLIP: Aligning Spike Video Streams with Natural Languagecited by 1
2 more
  1. 2026Exploring High-Order Self-Similarity for Video Understandingcited by 1
  2. 2025First-place Solution for Streetscape Shop Sign Recognition Competitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EV-CLIP: Efficient Visual Prompt Adaptation for CLIP in Few-shot Action Recognition under Visual Challenges
  2. 2025Is Temporal Prompting All We Need For Limited Labeled Action Recognition?
  3. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
  4. 2025MoCLIP-Lite: Efficient Video Recognition by Fusing CLIP with Motion Vectors
  5. 2025STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding
  6. 2025Omni-Video: Democratizing Unified Video Understanding and Generation
  7. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
  8. 2026Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation