# A Straightforward Framework For Video Retrieval Using CLIP

arXiv: https://arxiv.org/abs/2102.12443 · HTML: https://mlsanity.com/paper/2102.12443/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 24 Feb 2021.

## Builds on (most cited first, 14 of 14)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [A Dataset for Movie Description](https://mlsanity.com/paper/1501.02530/) (2015)
- [Support-set bottlenecks for video-text representation learning](https://mlsanity.com/paper/2010.02824/) (2020)
- [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020)
- [Learning a Text-Video Embedding from Incomplete and Heterogeneous Data](https://mlsanity.com/paper/1804.02516/) (2018)
- [Dual Encoding for Zero-Example Video Retrieval](https://mlsanity.com/paper/1809.06181/) (2018)
- [Predicting Visual Features from Text for Image and Video Caption Retrieval](https://mlsanity.com/paper/1709.01362/) (2017)
- [Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network](https://mlsanity.com/paper/1906.00377/) (2019)

## Built on by (most cited first, 30 of 66)

- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Simple but Effective: CLIP Embeddings for Embodied AI](https://mlsanity.com/paper/2111.09888/) (2021)
- [Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications](https://mlsanity.com/paper/2108.02818/) (2021)
- [X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval](https://mlsanity.com/paper/2203.15086/) (2022)
- [Bridging Video-text Retrieval with Multiple Choice Questions](https://mlsanity.com/paper/2201.04850/) (2022)
- [MDMMT: Multidomain Multimodal Transformer for Video Retrieval](https://mlsanity.com/paper/2103.10699/) (2021)
- [Disentangled Representation Learning for Text-Video Retrieval](https://mlsanity.com/paper/2203.07111/) (2022)
- [Verbs in Action: Improving verb understanding in video-language models](https://mlsanity.com/paper/2304.06708/) (2023)
- [CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations](https://mlsanity.com/paper/2109.14910/) (2021)
- [VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners](https://mlsanity.com/paper/2212.04979/) (2022)
- [UATVR: Uncertainty-Adaptive Text-Video Retrieval](https://mlsanity.com/paper/2301.06309/) (2023)
- [Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization](https://mlsanity.com/paper/2302.00624/) (2023)
- [Unified Coarse-to-Fine Alignment for Video-Text Retrieval](https://mlsanity.com/paper/2309.10091/) (2023)
- [ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound](https://mlsanity.com/paper/2204.02874/) (2022)
- [EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching](https://mlsanity.com/paper/2111.08919/) (2021)
- [HowToCaption: Prompting LLMs to Transform Video Annotations at Scale](https://mlsanity.com/paper/2310.04900/) (2023)
- [Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval](https://mlsanity.com/paper/2403.17998/) (2024)
- [Geodesic Multi-Modal Mixup for Robust Fine-Tuning](https://mlsanity.com/paper/2203.03897/) (2022)
- [MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval](https://mlsanity.com/paper/2204.12408/) (2022)
- [I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision](https://mlsanity.com/paper/2211.09778/) (2022)
- [Masking Modalities for Cross-modal Video Retrieval](https://mlsanity.com/paper/2111.01300/) (2021)
- [Lightweight Attentional Feature Fusion: A New Baseline for Text-to-Video Retrieval](https://mlsanity.com/paper/2112.01832/) (2021)
- [Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval](https://mlsanity.com/paper/2309.17093/) (2023)
- [Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Data](https://mlsanity.com/paper/2310.05010/) (2023)

## Abstract (arXiv)

Video Retrieval is a challenging task where a text query is matched to a video or vice versa. Most of the existing approaches for addressing such a problem rely on annotations made by the users. Although simple, this approach is not always feasible in practice. In this work, we explore the application of the language-image model, CLIP, to obtain video representations without the need for said annotations. This model was explicitly trained to learn a common space where images and text can be compared. Using various techniques described in this document, we extended its application to videos, obtaining state-of-the-art results on the MSR-VTT and MSVD benchmarks.