mlsanityData to 7 Oct 2026

CLIP2Video: Mastering Video-Text Retrieval via Image CLIP

arXiv 2106.11097 ↗First posted 21 Jun 2021cs.CVcited by 223

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present CLIP2Video network to transfer the image-language pre-training model to video-text retrieval in an end-to-end manner. Leading approaches in the domain of video-and-language learning try to distill the spatio-temporal video features and multi-modal interaction between videos and languages from a large-scale video-text dataset. Different from them, we leverage pretrained image-language model, simplify it as a two-stage framework with co-learning of image-text and enhancing temporal relations between video frames and video-text respectively, make it able to train on comparatively small datasets. Specifically, based on the spatial semantics captured by Contrastive Language-Image Pretraining (CLIP) model, our model involves a Temporal Difference Block to capture motions at fine temporal video frames, and a Temporal Alignment Block to re-align the tokens of video clips and phrases and enhance the multi-modal correlation. We conduct thorough ablation studies, and achieve state-of-the-art performance on major text-to-video and video-to-text retrieval benchmarks, including new records of retrieval accuracy on MSR-VTT, MSVD and VATEX.

Key facts. Cited by 223 papers in the map (data through 7 Oct 2026). Builds on 36 of its 44 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

36 of 44 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018SlowFast Networks for Video Recognitioncited by 1,862
  7. 2021ViViT: A Video Vision Transformercited by 1,610
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 935
  12. 2017Localizing Moments in Video with Natural Languagecited by 660
24 more
  1. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  2. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  3. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  4. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  5. 2020Multi-modal Transformer for Video Retrievalcited by 395
  6. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  7. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  8. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  9. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  10. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  11. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  12. 2020Support-set bottlenecks for video-text representation learningcited by 170
  13. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  14. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  15. 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
  16. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  17. 2018Dual Encoding for Zero-Example Video Retrievalcited by 85
  18. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  19. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74
  20. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  21. 2020Learning Video Representations from Textual Web Supervisioncited by 39
  22. 2020Learning Spatiotemporal Features via Video and Text Pair Discriminationcited by 38
  23. 2015The Long-Short Story of Movie Descriptioncited by 20
  24. 2018S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networkscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 50; 2023: 60; 2024: 56; 2025: 36; 2026: 8.

223 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  3. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  4. 2022Multimodal Learning with Transformers: A Surveycited by 399
  5. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  6. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  7. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  8. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  9. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  10. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  11. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  12. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
38 more
  1. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  2. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  3. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  4. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  5. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  6. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  7. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  9. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  10. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103
  11. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  12. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  13. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  14. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  15. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  16. 2022CLIP-Driven Fine-grained Text-Image Person Re-identificationcited by 79
  17. 2022Disentangled Representation Learning for Text-Video Retrievalcited by 77
  18. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  19. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  20. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  21. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  22. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  23. 2022CLIP model is an Efficient Continual Learnercited by 64
  24. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  25. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  26. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  27. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  28. 2023Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferringcited by 57
  29. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  30. 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55
  31. 2023DiffusionRet: Generative Text-Video Retrieval with Diffusion Modelcited by 52
  32. 2023Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Featurescited by 52
  33. 2023CoVR-2: Automatic Data Construction for Composed Video Retrievalcited by 52
  34. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  35. 2023Renderable Neural Radiance Map for Visual Navigationcited by 51
  36. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  37. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  38. 2024Foundation Models for Video Understanding: A Surveycited by 46

50 most cited of 223 All 223 (JSON)