# From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos

arXiv: https://arxiv.org/abs/2506.05274 · HTML: https://mlsanity.com/paper/2506.05274/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 53 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 30 of 40)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding](https://mlsanity.com/paper/2409.14485/) (2024)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [Zero-Shot Composed Image Retrieval with Textual Inversion](https://mlsanity.com/paper/2303.15247/) (2023)
- [GME: Improving Universal Multimodal Retrieval by Multimodal LLMs](https://mlsanity.com/paper/2412.16855/) (2024)
- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [AIM: Adapting Image Models for Efficient Video Action Recognition](https://mlsanity.com/paper/2302.03024/) (2023)
- [E5-V: Universal Embeddings with Multimodal Large Language Models](https://mlsanity.com/paper/2407.12580/) (2024)
- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training](https://mlsanity.com/paper/2301.02280/) (2023)
- [CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion](https://mlsanity.com/paper/2303.11916/) (2023)
- [A Zero-Shot Framework for Sketch-based Image Retrieval](https://mlsanity.com/paper/1807.11724/) (2018)
- [VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval](https://mlsanity.com/paper/2406.04292/) (2024)
- [Compositional Learning of Image-Text Query for Image Retrieval](https://mlsanity.com/paper/2006.11149/) (2020)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [Data Roaming and Quality Assessment for Composed Image Retrieval](https://mlsanity.com/paper/2303.09429/) (2023)
- [Video Pose Distillation for Few-Shot, Fine-Grained Sports Action Recognition](https://mlsanity.com/paper/2109.01305/) (2021)
- [Composed Video Retrieval via Enriched Context and Discriminative Embeddings](https://mlsanity.com/paper/2403.16997/) (2024)
- [Vision Transformers for Action Recognition: A Survey](https://mlsanity.com/paper/2209.05700/) (2022)

## Built on by (most cited first, 4 of 4)

- [STROKEVISION-BENCH: A Multimodal Video And 2D Pose Benchmark For Tracking Stroke Recovery](https://mlsanity.com/paper/2509.07994/) (2025)
- [Controlling Embedding Spaces with Text-Conditioned Transformations](https://mlsanity.com/paper/2607.22919/) (2026)
- [A Survey of Large Models in Sports](https://mlsanity.com/paper/2608.14377/) (2026)
- [Beyond Similarity: Foundation Models as an Efficient Backbone for Training-Free Composed Video Retrieval](https://mlsanity.com/paper/2609.10008/) (2026)

## Abstract (arXiv)

Composed Video Retrieval (CoVR) retrieves a target video given a query video and a modification text describing the intended change. Existing CoVR benchmarks emphasize appearance shifts or coarse event changes and therefore do not test the ability to capture subtle, fast-paced temporal differences. We introduce TF-CoVR, the first large-scale benchmark dedicated to temporally fine-grained CoVR. TF-CoVR focuses on gymnastics and diving, and provides 180K triplets drawn from FineGym and FineDiving datasets. Previous CoVR benchmarks, focusing on temporal aspect, link each query to a single target segment taken from the same video, limiting practical usefulness. In TF-CoVR, we instead construct each <query, modification> pair by prompting an LLM with the label differences between clips drawn from different videos; every pair is thus associated with multiple valid target videos (3.9 on average), reflecting real-world tasks such as sports-highlight generation. To model these temporal dynamics, we propose TF-CoVR-Base, a concise two-stage training framework: (i) pre-train a video encoder on fine-grained action classification to obtain temporally discriminative embeddings; (ii) align the composed query with candidate videos using contrastive learning. We conduct the first comprehensive study of image, video, and general multimodal embedding (GME) models on temporally fine-grained composed retrieval in both zero-shot and fine-tuning regimes. On TF-CoVR, TF-CoVR-Base improves zero-shot mAP@50 from 5.92 (LanguageBind) to 7.51, and after fine-tuning raises the state-of-the-art from 19.83 to 27.22.