mlsanityData to 7 Oct 2026

Frozen CLIP Models are Efficient Video Learners

arXiv 2208.03550 ↗First posted 6 Aug 2022cs.CVcited by 181

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video recognition has been dominated by the end-to-end learning paradigm -- first initializing a video recognition model with weights of a pretrained image model and then conducting end-to-end training on videos. This enables the video network to benefit from the pretrained image model. However, this requires substantial computation and memory resources for finetuning on videos and the alternative of directly using pretrained image features without finetuning the image backbone leads to subpar results. Fortunately, recent advances in Contrastive Vision-Language Pre-training (CLIP) pave the way for a new route for visual recognition tasks. Pretrained on large open-vocabulary image-text pair data, these models learn powerful visual representations with rich semantics. In this paper, we present Efficient Video Learning (EVL) -- an efficient framework for directly training high-quality video recognition models with frozen CLIP features. Specifically, we employ a lightweight Transformer decoder and learn a query token to dynamically collect frame-level spatial features from the CLIP image encoder. Furthermore, we adopt a local temporal module in each decoder layer to discover temporal clues from adjacent frames and their attention maps. We show that despite being efficient to train with a frozen backbone, our models learn high quality video representations on a variety of video recognition datasets. Code is available at https://github.com/OpenGVLab/efficient-video-recognition.

Key facts. Cited by 181 papers in the map (data through 7 Oct 2026). Builds on 56 of its 62 references that are in the map. First posted on arXiv 6 Aug 2022.

Builds on

56 of 62 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
38 more
  1. 2017Non-local Neural Networkscited by 2,290
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  3. 2018SlowFast Networks for Video Recognitioncited by 1,862
  4. 2021ViViT: A Video Vision Transformercited by 1,610
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  7. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  8. 2021Video Swin Transformercited by 1,090
  9. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  11. 2021Scaling Vision Transformerscited by 993
  12. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  13. 2021Multiscale Vision Transformerscited by 896
  14. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  15. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  16. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  17. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  18. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  19. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  20. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  21. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  22. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  23. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  24. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  25. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  26. 2017Temporal Relational Reasoning in Videoscited by 324
  27. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  28. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  29. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  30. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  31. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  32. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  33. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  34. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  35. 2017Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognitioncited by 178
  36. 2022Multiview Transformers for Video Recognitioncited by 148
  37. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  38. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 63; 2024: 55; 2025: 31; 2026: 19.

181 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  4. 2023Personalize Segment Anything Model with One Shotcited by 250
  5. 2022Learning Video Representations from Large Language Modelscited by 204
  6. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  7. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  8. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  9. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  10. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  11. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  12. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
38 more
  1. 2023VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detectioncited by 121
  2. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  3. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  4. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 90
  5. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  6. 2024Streaming Dense Video Captioningcited by 82
  7. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  8. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  9. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  10. 2023Diffusion Models as Masked Autoencoderscited by 71
  11. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  12. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  13. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  14. 2024Endora: Video Generation Models as Endoscopy Simulatorscited by 53
  15. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
  16. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  17. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
  18. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 48
  19. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 44
  20. 2023Annotation-free Audio-Visual Segmentationcited by 44
  21. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  22. 2023CLIP-guided Prototype Modulating for Few-shot Action Recognitioncited by 41
  23. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  24. 2023Dual-path Adaptation from Image to Video Transformerscited by 40
  25. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptorscited by 39
  26. 2023VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Biascited by 37
  27. 2024Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Modelcited by 37
  28. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  29. 2023Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokenscited by 35
  30. 2023Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysiscited by 34
  31. 2024R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Groundingcited by 34
  32. 2023EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understandingcited by 33
  33. 2024WATT: Weight Average Test-Time Adaptation of CLIPcited by 33
  34. 2023Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Trainingcited by 32
  35. 2023Learning Emotion Representations from Verbal and Nonverbal Communicationcited by 31
  36. 2023Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learningcited by 31
  37. 2023EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognitioncited by 31
  38. 2023ViT-Lens: Towards Omni-modal Representationscited by 29

50 most cited of 181 All 181 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
  2. 2025MoCLIP-Lite: Efficient Video Recognition by Fusing CLIP with Motion Vectors
  3. 2025From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understanding
  4. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
  5. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  6. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
  7. 2025Is Temporal Prompting All We Need For Limited Labeled Action Recognition?
  8. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models