# Finding Moments in Video Collections Using Natural Language

arXiv: https://arxiv.org/abs/1907.12763 · HTML: https://mlsanity.com/paper/1907.12763/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2019.

## Built on by (most cited first, 10 of 10)

- [Beyond Uncertainty: Evidential Deep Learning for Robust Video Temporal Grounding](https://mlsanity.com/paper/2408.16272/) (2024)
- [LD-DETR: Loop Decoder DEtection TRansformer for Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2501.10787/) (2025)
- [CVA: Context-aware Video-text Alignment for Video Temporal Grounding](https://mlsanity.com/paper/2603.24934/) (2026)
- [VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval](https://mlsanity.com/paper/2412.01558/) (2024)
- [Text-Video Multi-Grained Integration for Video Moment Montage](https://mlsanity.com/paper/2412.09276/) (2024)
- [The Devil is in the Spurious Correlations: Boosting Moment Retrieval with Dynamic Learning](https://mlsanity.com/paper/2501.07305/) (2025)
- [DiffusionVMR: Diffusion Model for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2308.15109/) (2023)
- [TVR-Ranking: A Dataset for Ranked Video Moment Retrieval with Imprecise Queries](https://mlsanity.com/paper/2407.06597/) (2024)
- [ResidualViT for Efficient Temporally Dense Video Encoding](https://mlsanity.com/paper/2509.13255/) (2025)
- [ClipTBP: Clip-Pair based Temporal Boundary Prediction with Boundary-Aware Learning for Moment Retrieval](https://mlsanity.com/paper/2604.27591/) (2026)

## Abstract (arXiv)

We introduce the task of retrieving relevant video moments from a large corpus of untrimmed, unsegmented videos given a natural language query. Our task poses unique challenges as a system must efficiently identify both the relevant videos and localize the relevant moments in the videos. To address these challenges, we propose SpatioTemporal Alignment with Language (STAL), a model that represents a video moment as a set of regions within a series of short video clips and aligns a natural language query to the moment's regions. Our alignment cost compares variable-length language and video features using symmetric squared Chamfer distance, which allows for efficient indexing and retrieval of the video moments. Moreover, aligning language features to regions within a video moment allows for finer alignment compared to methods that extract only an aggregate feature from the entire video moment. We evaluate our approach on two recently proposed datasets for temporal localization of moments in video with natural language (DiDeMo and Charades-STA) extended to our video corpus moment retrieval setting. We show that our STAL re-ranking model outperforms the recently proposed Moment Context Network on all criteria across all datasets on our proposed task, obtaining relative gains of 37% - 118% for average recall and up to 30% for median rank. Moreover, our approach achieves more than 130x faster retrieval and 8x smaller index size with a 1M video corpus in an approximate setting.