mlsanityData to 7 Oct 2026

CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval

arXiv 2104.08860 ↗First posted 18 Apr 2021cs.CVcited by 645

Not on the map. Why?

Cited by 84 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video-text retrieval plays an essential role in multi-modal research and has been widely used in many real-world web applications. The CLIP (Contrastive Language-Image Pre-training), an image-language pre-training model, has demonstrated the power of visual concepts learning from web collected image-text datasets. In this paper, we propose a CLIP4Clip model to transfer the knowledge of the CLIP model to video-language retrieval in an end-to-end manner. Several questions are investigated via empirical studies: 1) Whether image feature is enough for video-text retrieval? 2) How a post-pretraining on a large-scale video-text dataset based on the CLIP affect the performance? 3) What is the practical mechanism to model temporal dependency between video frames? And 4) The Hyper-parameters sensitivity of the model on video-text retrieval task. Extensive experimental results present that the CLIP4Clip model transferred from the CLIP can achieve SOTA results on various video-text retrieval datasets, including MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo. We release our code at https://github.com/ArrowLuo/CLIP4Clip.

Key facts. Cited by 645 papers in the map (data through 7 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

38 of 52 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
26 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  2. 2017Dense-Captioning Events in Videoscited by 789
  3. 2017Localizing Moments in Video with Natural Languagecited by 660
  4. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  5. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  6. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  7. 2020Multi-modal Transformer for Video Retrievalcited by 395
  8. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  9. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  10. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  11. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  12. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  13. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  14. 2020Support-set bottlenecks for video-text representation learningcited by 170
  15. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  16. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  17. 2014Translating Videos to Natural Language Using Deep Recurrent Neural Networkscited by 95
  18. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  19. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  20. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  21. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74
  22. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  23. 2016End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answeringcited by 60
  24. 2015Video Paragraph Captioning Using Hierarchical Recurrent Neural Networkscited by 54
  25. 2016Learning Language-Visual Embedding for Movie Understanding with Natural-Languagecited by 41
  26. 2015The Long-Short Story of Movie Descriptioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 101; 2023: 158; 2024: 129; 2025: 148; 2026: 78.

645 papers.

Sort:

By area:

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  3. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  8. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  9. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  10. 2022Multimodal Learning with Transformers: A Surveycited by 399
  11. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  12. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
38 more
  1. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  2. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  4. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  5. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  6. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  7. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  8. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  9. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  10. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  11. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
  12. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  13. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  14. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  15. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  16. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  17. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  18. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  19. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  20. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  21. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  22. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  23. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  24. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  25. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  26. 2023VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detectioncited by 121
  27. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  28. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  29. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  30. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  31. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  32. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  33. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  34. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 107
  35. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  36. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  37. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  38. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94

50 most cited of 645 All 645 (JSON)