mlsanityData to 14 Oct 2026

FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Tasks

arXiv 2203.13371 ↗First posted 24 Mar 2022cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Large-scale pretrained image-text models have shown incredible zero-shot performance in a handful of tasks, including video ones such as action recognition and text-to-video retrieval. However, these models have not been adapted to video, mainly because they do not account for the time dimension but also because video frames are different from the typical images (e.g., containing motion blur, and less sharpness). In this paper, we present a fine-tuning strategy to refine these large-scale pretrained image-text models for zero-shot video understanding tasks. We show that by carefully adapting these models we obtain considerable improvements on two zero-shot Action Recognition tasks and three zero-shot Text-to-video Retrieval tasks. The code is available at https://github.com/bryant1410/fitclip

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 49 of its 75 references that are in the map. First posted on arXiv 24 Mar 2022.

Builds on

49 of 75 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  11. 2021ViViT: A Video Vision Transformercited by 1,620
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
37 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  2. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  3. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  4. 2021Robust fine-tuning of zero-shot modelscited by 797
  5. 2017Localizing Moments in Video with Natural Languagecited by 663
  6. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  7. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  8. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  9. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  10. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  11. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  12. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  13. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  14. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  15. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  16. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  17. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  18. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  19. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  20. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  21. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  22. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  23. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  24. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  25. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  26. 2020Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applicationscited by 80
  27. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  28. 2021Elaborative Rehearsal for Zero-shot Action Recognitioncited by 77
  29. 2020SMART Frame Selection for Action Recognitioncited by 73
  30. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  31. 2019Out-of-Distribution Detection for Generalized Zero-Shot Action Recognitioncited by 54
  32. 2015Objects2action: Classifying and localizing actions without any video examplecited by 26
  33. 2019DistInit: Learning Video Representations Without a Single Labeled Videocited by 21
  34. 2019Zero-Shot Action Recognition in Videos: A Surveycited by 19
  35. 2018S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networkscited by 16
  36. 2020ZSTAD: Zero-Shot Temporal Activity Detectioncited by 16
  37. 2021A New Split for Evaluating True Zero-Shot Action Recognitioncited by 14

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 1; 2024: 4; 2025: 6.

16 papers.

Sort:

By area:

  1. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  2. 2025Vision-Language Models Do Not Understand Negationcited by 72
  3. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 70
  4. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  5. 2022PIVOT: Prompting for Video Continual Learningcited by 45
  6. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
  7. 2024Diffusion Feedback Helps CLIP See Bettercited by 38
  8. 2022Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Surveycited by 12
  9. 2024CLoVe: Encoding Compositional Language in Contrastive Vision-Language Modelscited by 12
  10. 2025Learning to Generalize without Bias for Open-Vocabulary Action Recognitioncited by 7
  11. 2024Learning text-to-video retrieval from image captioningcited by 6
  12. 2025Pose-Aware Weakly-Supervised Action Segmentationcited by 2
4 more
  1. 2025ResidualViT for Efficient Temporally Dense Video Encodingcited by 2
  2. 2025Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Surveycited by 2
  3. 2022REST: REtrieve & Self-Train for generative action recognitioncited by 1
  4. 2025Advancing Video Self-Supervised Learning via Image Foundation Modelscited by 1