# Condensed Movies: Story Based Retrieval with Contextual Embeddings

arXiv: https://arxiv.org/abs/2005.04208 · HTML: https://mlsanity.com/paper/2005.04208/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 30 of 95)

- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens](https://mlsanity.com/paper/2404.03413/) (2024)
- [CinePile: A Long Video Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2405.08813/) (2024)
- [Long Movie Clip Classification with State-Space Video Models](https://mlsanity.com/paper/2204.01692/) (2022)
- [MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence](https://mlsanity.com/paper/2407.16655/) (2024)
- [Learning Audio-Video Modalities from Image Captions](https://mlsanity.com/paper/2204.00679/) (2022)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning](https://mlsanity.com/paper/2210.06031/) (2022)
- [A CLIP-Hitchhiker's Guide to Long Video Retrieval](https://mlsanity.com/paper/2205.08508/) (2022)
- [AutoAD: Movie Description in Context](https://mlsanity.com/paper/2303.16899/) (2023)
- [AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description](https://mlsanity.com/paper/2310.06838/) (2023)
- [TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding](https://mlsanity.com/paper/2310.19060/) (2023)
- [On Semantic Similarity in Video Retrieval](https://mlsanity.com/paper/2103.10095/) (2021)
- [Goldfish: Vision-Language Understanding of Arbitrarily Long Videos](https://mlsanity.com/paper/2407.12679/) (2024)
- [AutoAD III: The Prequel -- Back to the Pixels](https://mlsanity.com/paper/2404.14412/) (2024)
- [Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding](https://mlsanity.com/paper/2502.06020/) (2025)
- [MovieCuts: A New Dataset and Benchmark for Cut Type Recognition](https://mlsanity.com/paper/2109.05569/) (2021)
- [The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing](https://mlsanity.com/paper/2207.09812/) (2022)
- [E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness](https://mlsanity.com/paper/2407.01516/) (2024)
- [Face, Body, Voice: Video Person-Clustering with Multiple Modalities](https://mlsanity.com/paper/2105.09939/) (2021)
- [Masking Modalities for Cross-modal Video Retrieval](https://mlsanity.com/paper/2111.01300/) (2021)
- [AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description](https://mlsanity.com/paper/2407.15850/) (2024)
- [Hierarchical Self-supervised Representation Learning for Movie Understanding](https://mlsanity.com/paper/2204.03101/) (2022)
- [Learning to Cut by Watching Movies](https://mlsanity.com/paper/2108.04294/) (2021)
- [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024)
- [MovieCLIP: Visual Scene Recognition in Movies](https://mlsanity.com/paper/2210.11065/) (2022)
- [Learning Video Context as Interleaved Multimodal Sequences](https://mlsanity.com/paper/2407.21757/) (2024)
- [Audio-Sync Video Generation with Multi-Stream Temporal Control](https://mlsanity.com/paper/2506.08003/) (2025)
- [Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation](https://mlsanity.com/paper/2504.01020/) (2025)

## Abstract (arXiv)

Our objective in this work is long range understanding of the narrative structure of movies. Instead of considering the entire movie, we propose to learn from the `key scenes' of the movie, providing a condensed look at the full storyline. To this end, we make the following three contributions: (i) We create the Condensed Movies Dataset (CMD) consisting of the key scenes from over 3K movies: each key scene is accompanied by a high level semantic description of the scene, character face-tracks, and metadata about the movie. The dataset is scalable, obtained automatically from YouTube, and is freely available for anybody to download and use. It is also an order of magnitude larger than existing movie datasets in the number of movies; (ii) We provide a deep network baseline for text-to-video retrieval on our dataset, combining character, speech and visual cues into a single video embedding; and finally (iii) We demonstrate how the addition of context from other video clips improves retrieval performance.