mlsanityData to 7 Oct 2026

Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization

arXiv 2302.00624 ↗First posted 1 Feb 2023cs.CVcited by 54

Not on the map. Why?

Abstract (arXiv)

Contrastive Language-Image Pretraining (CLIP) has demonstrated impressive zero-shot learning abilities for image understanding, yet limited effort has been made to investigate CLIP for zero-shot video recognition. We introduce Open-VCLIP, a simple yet effective approach that transforms CLIP into a strong zero-shot video classifier that can recognize unseen actions and events at test time. Our framework extends CLIP with minimal modifications to model spatial-temporal relationships in videos, making it a specialized video classifier, while striving for generalization. We formally show that training an Open-VCLIP is equivalent to continual learning with zero historical data. To address this problem, we propose Interpolated Weight Optimization, which utilizes the benefit of weight interpolation in both training and test time. We evaluate our method on three popular and challenging action recognition datasets following various zero-shot evaluation protocols and we demonstrate our approach outperforms state-of-the-art methods by clear margins. In particular, we achieve 87.9%, 58.3%, 81.1% zero-shot accuracy on UCF, HMDB and Kinetics-600 respectively, outperforming state-of-the-art methods by 8.3%, 7.8% and 12.2%. Code is released at https://github.com/wengzejia1/Open-VCLIP.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

38 of 46 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  10. 2017Continual Learning with Deep Generative Replaycited by 1,152
  11. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  12. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
26 more
  1. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  2. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  3. 2018A Short Note about Kinetics-600cited by 331
  4. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  5. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  6. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  7. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  8. 2022Patching open-vocabulary models by interpolating weightscited by 187
  9. 2021Space-time Mixing Attention for Video Transformercited by 84
  10. 2020Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applicationscited by 79
  11. 2021Elaborative Rehearsal for Zero-shot Action Recognitioncited by 76
  12. 2019TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognitioncited by 71
  13. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 70
  14. 2022SVFormer: Semi-supervised Video Transformer for Action Recognitioncited by 67
  15. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  16. 2022CLIP model is an Efficient Continual Learnercited by 64
  17. 2019Out-of-Distribution Detection for Generalized Zero-Shot Action Recognitioncited by 54
  18. 2018A Generative Approach to Zero-Shot and Few-Shot Action Recognitioncited by 39
  19. 2015Transductive Zero-Shot Action Recognition by Word-Vector Embeddingcited by 31
  20. 2015Objects2action: Classifying and localizing actions without any video examplecited by 26
  21. 2020The Effectiveness of Memory Replay in Large Scale Continual Learningcited by 26
  22. 2017Zero-Shot Activity Recognition with Verb Attribute Inductioncited by 15
  23. 2022Cross-modal Representation Learning for Zero-shot Action Recognitioncited by 15
  24. 2022Bridging CLIP and StyleGAN through Latent Alignment for Image Editingcited by 12
  25. 2021One Pass ImageNetcited by 11
  26. 2020Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpointcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 21; 2025: 13; 2026: 11.

54 papers.

Sort:

By area:

  1. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  2. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 44
  3. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  4. 2024FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognitioncited by 35
  5. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  6. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  7. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 23
  8. 2023Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Datacited by 21
  9. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Predictioncited by 20
  10. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Modelscited by 19
  11. 2023HairCLIPv2: Unifying Hair Editing via Proxy Feature Blendingcited by 17
  12. 20233DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Modelscited by 17
38 more
  1. 2024XOV-Action: Towards Generalizable Open-Vocabulary Action Recognitioncited by 15
  2. 2024OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learningcited by 14
  3. 2025Anomize: Better Open Vocabulary Video Anomaly Detectioncited by 13
  4. 2023Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Promptingcited by 12
  5. 2025Modeling Multiple Normal Action Representations for Error Detection in Procedural Taskscited by 12
  6. 2023Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detectioncited by 11
  7. 2024GenRec: Unifying Video Generation and Recognition with Diffusion Modelscited by 11
  8. 2024Leveraging Temporal Contextualization for Video Action Recognitioncited by 9
  9. 2024Efficient Test-Time Prompt Tuning for Vision-Language Modelscited by 9
  10. 2024Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIPcited by 9
  11. 2025SCA3D: Enhancing Cross-modal 3D Retrieval via 3D Shape and Caption Paired Data Augmentationcited by 9
  12. 2024Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detectioncited by 8
  13. 2023Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognitioncited by 7
  14. 2025Learning to Generalize without Bias for Open-Vocabulary Action Recognitioncited by 7
  15. 2024Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localizationcited by 6
  16. 2024Open-Vocabulary Spatio-Temporal Action Detectioncited by 5
  17. 2024Open Vocabulary Multi-Label Video Classificationcited by 5
  18. 2024MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfercited by 5
  19. 2024Efficient Transfer Learning for Video-language Foundation Modelscited by 4
  20. 2024Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution Viewcited by 3
  21. 2024End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Promptingcited by 3
  22. 2025CueBench: Advancing Unified Understanding of Context-Aware Video Anomalies in Real-Worldcited by 3
  23. 2025VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Modelscited by 3
  24. 2025The Role of Video Generation in Enhancing Data-Limited Action Understandingcited by 2
  25. 2024Skim then Focus: Integrating Contextual and Fine-grained Views for Repetitive Action Countingcited by 1
  26. 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  27. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Modelscited by 1
  28. 2026DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Datasetcited by 1
  29. 2026EduVQA: Towards Concept-Aware Assessment of Educational AI-Generated Videoscited by 1
  30. 2024The impact of Compositionality in Zero-shot Multi-label action recognition for Object-based taskscited by 0
  31. 2024EOV-Seg: Efficient Open-Vocabulary Panoptic Segmentationcited by 0
  32. 2025Can You Trust What You See? Alpha Channel No-Box Attacks on Video Object Detectioncited by 0
  33. 2025Text-guided Weakly Supervised Framework for Dynamic Facial Expression Recognitioncited by 0
  34. 2026ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understandingcited by 0
  35. 2026EV-CLIP: Efficient Visual Prompt Adaptation for CLIP in Few-shot Action Recognition under Visual Challengescited by 0
  36. 2026Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognitioncited by 0
  37. 2026MAIL++: Multi-Modal Bi-directional Agent Layer for Vision-Language Modelscited by 0
  38. 2026Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmeticcited by 0

50 most cited of 54 All 54 (JSON)