# An Empirical Study of Frame Selection for Text-to-Video Retrieval

arXiv: https://arxiv.org/abs/2311.00298 · HTML: https://mlsanity.com/paper/2311.00298/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Revisiting the "Video" in Video-Language Understanding](https://mlsanity.com/paper/2206.01720/) (2022)
- [Align and Prompt: Video-and-Language Pre-training with Entity Prompts](https://mlsanity.com/paper/2112.09583/) (2021)
- [X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval](https://mlsanity.com/paper/2203.15086/) (2022)
- [Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss](https://mlsanity.com/paper/2109.04290/) (2021)
- [TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval](https://mlsanity.com/paper/2207.07852/) (2022)
- [Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?](https://mlsanity.com/paper/2301.00184/) (2022)
- [UATVR: Uncertainty-Adaptive Text-Video Retrieval](https://mlsanity.com/paper/2301.06309/) (2023)
- [Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning](https://mlsanity.com/paper/2303.14369/) (2023)
- [Tagging before Alignment: Integrating Multi-Modal Tags for Video-Text Retrieval](https://mlsanity.com/paper/2301.12644/) (2023)
- [LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling](https://mlsanity.com/paper/2210.11929/) (2022)
- [SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-training](https://mlsanity.com/paper/2211.11446/) (2022)
- [Learning Trajectory-Word Alignments for Video-Language Tasks](https://mlsanity.com/paper/2301.01953/) (2023)
- [Contrastive Video-Language Learning with Fine-grained Frame Sampling](https://mlsanity.com/paper/2210.05039/) (2022)
- [Efficient Cross-Modal Video Retrieval with Meta-Optimized Frames](https://mlsanity.com/paper/2210.08452/) (2022)

## Built on by (most cited first, 6 of 6)

- [Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions](https://mlsanity.com/paper/2503.05186/) (2025)
- [Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmark](https://mlsanity.com/paper/2412.08879/) (2024)
- [A Lightweight Moment Retrieval System with Global Re-Ranking and Robust Adaptive Bidirectional Temporal Search](https://mlsanity.com/paper/2504.09298/) (2025)
- [Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering](https://mlsanity.com/paper/2410.09380/) (2024)
- [Towards Scalable Video Anomaly Retrieval: A Synthetic Video-Text Benchmark](https://mlsanity.com/paper/2506.01466/) (2025)
- [Shot-Aware Frame Sampling for Video Understanding](https://mlsanity.com/paper/2603.17374/) (2026)

## Abstract (arXiv)

Text-to-video retrieval (TVR) aims to find the most relevant video in a large video gallery given a query text. The intricate and abundant context of the video challenges the performance and efficiency of TVR. To handle the serialized video contexts, existing methods typically select a subset of frames within a video to represent the video content for TVR. How to select the most representative frames is a crucial issue, whereby the selected frames are required to not only retain the semantic information of the video but also promote retrieval efficiency by excluding temporally redundant frames. In this paper, we make the first empirical study of frame selection for TVR. We systemically classify existing frame selection methods into text-free and text-guided ones, under which we detailedly analyze six different frame selections in terms of effectiveness and efficiency. Among them, two frame selections are first developed in this paper. According to the comprehensive analysis on multiple TVR benchmarks, we empirically conclude that the TVR with proper frame selections can significantly improve the retrieval efficiency without sacrificing the retrieval performance.