# Exploiting Temporal Relationships in Video Moment Localization with Natural Language

arXiv: https://arxiv.org/abs/1908.03846 · HTML: https://mlsanity.com/paper/1908.03846/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Aug 2019.

## Built on by (most cited first, 30 of 33)

- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [Video Corpus Moment Retrieval with Contrastive Learning](https://mlsanity.com/paper/2105.06247/) (2021)
- [Natural Language Video Localization: A Revisit in Span-based Question Answering Framework](https://mlsanity.com/paper/2102.13558/) (2021)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval](https://mlsanity.com/paper/2109.10016/) (2021)
- [Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding](https://mlsanity.com/paper/2109.06400/) (2021)
- [Correlation-Guided Query-Dependency Calibration for Video Temporal Grounding](https://mlsanity.com/paper/2311.08835/) (2023)
- [Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval](https://mlsanity.com/paper/2407.15051/) (2024)
- [Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding](https://mlsanity.com/paper/2103.16848/) (2021)
- [A Survey on Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2109.08039/) (2021)
- [BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2312.00083/) (2023)
- [Relation-aware Video Reading Comprehension for Temporal Language Grounding](https://mlsanity.com/paper/2110.05717/) (2021)
- [Video Activity Localisation with Uncertainties in Temporal Boundary](https://mlsanity.com/paper/2206.12923/) (2022)
- [Parallel Attention Network with Sequence Matching for Video Grounding](https://mlsanity.com/paper/2105.08481/) (2021)
- [A Survey on Video Moment Localization](https://mlsanity.com/paper/2306.07515/) (2023)
- [Progressive Localization Networks for Language-based Moment Localization](https://mlsanity.com/paper/2102.01282/) (2021)
- [Video-aided Unsupervised Grammar Induction](https://mlsanity.com/paper/2104.04369/) (2021)
- [MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval](https://mlsanity.com/paper/2406.17880/) (2024)
- [Learning Sample Importance for Cross-Scenario Video Temporal Grounding](https://mlsanity.com/paper/2201.02848/) (2022)
- [Prototypes are Balanced Units for Efficient and Effective Partially Relevant Video Retrieval](https://mlsanity.com/paper/2504.13035/) (2025)
- [Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understanding](https://mlsanity.com/paper/2410.13598/) (2024)
- [A Survey on Natural Language Video Localization](https://mlsanity.com/paper/2104.00234/) (2021)
- [Diversifying Query: Region-Guided Transformer for Temporal Sentence Grounding](https://mlsanity.com/paper/2406.00143/) (2024)
- [Deep Understanding of Sign Language for Sign to Subtitle Alignment](https://mlsanity.com/paper/2503.03287/) (2025)
- [Vid-Morp: Video Moment Retrieval Pretraining from Unlabeled Videos in the Wild](https://mlsanity.com/paper/2412.00811/) (2024)
- [SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses](https://mlsanity.com/paper/2408.01669/) (2024)
- [Sim-DETR: Unlock DETR for Temporal Sentence Grounding](https://mlsanity.com/paper/2509.23867/) (2025)
- [Learning a Grammar Inducer from Massive Uncurated Instructional Videos](https://mlsanity.com/paper/2210.12309/) (2022)
- [Language-Conditioned Change-point Detection to Identify Sub-Tasks in Robotics Domains](https://mlsanity.com/paper/2309.00743/) (2023)

## Abstract (arXiv)

We address the problem of video moment localization with natural language, i.e. localizing a video segment described by a natural language sentence. While most prior work focuses on grounding the query as a whole, temporal dependencies and reasoning between events within the text are not fully considered. In this paper, we propose a novel Temporal Compositional Modular Network (TCMN) where a tree attention network first automatically decomposes a sentence into three descriptions with respect to the main event, context event and temporal signal. Two modules are then utilized to measure the visual similarity and location similarity between each segment and the decomposed descriptions. Moreover, since the main event and context event may rely on different modalities (RGB or optical flow), we use late fusion to form an ensemble of four models, where each model is independently trained by one combination of the visual input. Experiments show that our model outperforms the state-of-the-art methods on the TEMPO dataset.