# A Survey on Temporal Sentence Grounding in Videos

arXiv: https://arxiv.org/abs/2109.08039 · HTML: https://mlsanity.com/paper/2109.08039/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 145 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 30 of 80)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Dense Regression Network for Video Grounding](https://mlsanity.com/paper/2004.03545/) (2020)
- [To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression](https://mlsanity.com/paper/1804.07014/) (2018)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment](https://mlsanity.com/paper/1812.00087/) (2018)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Multilevel Language and Vision Integration for Text-to-Clip Retrieval](https://mlsanity.com/paper/1804.05113/) (2018)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Single Shot Temporal Action Detection](https://mlsanity.com/paper/1710.06236/) (2017)
- [Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences](https://mlsanity.com/paper/2001.06891/) (2020)

## Built on by (most cited first, 29 of 29)

- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video](https://mlsanity.com/paper/2203.06667/) (2022)
- [Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges](https://mlsanity.com/paper/2309.13925/) (2023)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directions](https://mlsanity.com/paper/2406.05785/) (2024)
- [A Survey on Video Temporal Grounding with Multimodal Large Language Model](https://mlsanity.com/paper/2508.10922/) (2025)
- [Text-Visual Prompting for Efficient 2D Temporal Video Grounding](https://mlsanity.com/paper/2303.04995/) (2023)
- [Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols](https://mlsanity.com/paper/2311.02538/) (2023)
- [Grounding-Prompter: Prompting LLM with Multimodal Information for Temporal Sentence Grounding in Long Videos](https://mlsanity.com/paper/2312.17117/) (2023)
- [Learning to Locate Visual Answer in Video Corpus Using Question](https://mlsanity.com/paper/2210.05423/) (2022)
- [VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding](https://mlsanity.com/paper/2410.08593/) (2024)
- [Temporal Sentence Grounding in Streaming Videos](https://mlsanity.com/paper/2308.07102/) (2023)
- [LLM4VG: Large Language Models Evaluation for Video Grounding](https://mlsanity.com/paper/2312.14206/) (2023)
- [Counterfactual Cross-modality Reasoning for Weakly Supervised Video Moment Localization](https://mlsanity.com/paper/2308.05648/) (2023)
- [A Simple Transformer-Based Model for Ego4D Natural Language Queries Challenge](https://mlsanity.com/paper/2211.08704/) (2022)
- [Cross-modal Contrastive Learning with Asymmetric Co-attention Network for Video Moment Retrieval](https://mlsanity.com/paper/2312.07435/) (2023)
- [Affordance-First Decomposition for Continual Learning in Video-Language Understanding](https://mlsanity.com/paper/2512.00694/) (2025)
- [Transform-Equivariant Consistency Learning for Temporal Sentence Grounding](https://mlsanity.com/paper/2305.04123/) (2023)
- [UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark](https://mlsanity.com/paper/2410.01180/) (2024)
- [Learning to Unify Audio, Visual and Text for Audio-Enhanced Multilingual Visual Answer Localization](https://mlsanity.com/paper/2411.02851/) (2024)
- [SD-VSum: A Method and Dataset for Script-Driven Video Summarization](https://mlsanity.com/paper/2505.03319/) (2025)
- [FedVMR: A New Federated Learning method for Video Moment Retrieval](https://mlsanity.com/paper/2210.15977/) (2022)
- [Multi-sentence Video Grounding for Long Video Generation](https://mlsanity.com/paper/2407.13219/) (2024)
- [Video sentence grounding with temporally global textual knowledge](https://mlsanity.com/paper/2404.13611/) (2024)
- [Collaborative Temporal Consistency Learning for Point-supervised Natural Language Video Localization](https://mlsanity.com/paper/2503.17651/) (2025)
- [GenSpan: Generation-Calibrated Motion Span Priors for Multi-Verb Video Corpus Moment Retrieval](https://mlsanity.com/paper/2603.22121/) (2026)
- [Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis](https://mlsanity.com/paper/2605.00826/) (2026)
- [FAST: A Holistic Framework for Optimizing Memory-I/O, Computation, and Sampling in Temporal GNN Training](https://mlsanity.com/paper/2607.05095/) (2026)

## Abstract (arXiv)

Temporal sentence grounding in videos(TSGV), which aims to localize one target segment from an untrimmed video with respect to a given sentence query, has drawn increasing attentions in the research community over the past few years. Different from the task of temporal action localization, TSGV is more flexible since it can locate complicated activities via natural languages, without restrictions from predefined action categories. Meanwhile, TSGV is more challenging since it requires both textual and visual understanding for semantic alignment between two modalities(i.e., text and video). In this survey, we give a comprehensive overview for TSGV, which i) summarizes the taxonomy of existing methods, ii) provides a detailed description of the evaluation protocols(i.e., datasets and metrics) to be used in TSGV, and iii) in-depth discusses potential problems of current benchmarking designs and research directions for further investigations. To the best of our knowledge, this is the first systematic survey on temporal sentence grounding. More specifically, we first discuss existing TSGV approaches by grouping them into four categories, i.e., two-stage methods, end-to-end methods, reinforcement learning-based methods, and weakly supervised methods. Then we present the benchmark datasets and evaluation metrics to assess current research progress. Finally, we discuss some limitations in TSGV through pointing out potential problems improperly resolved in the current evaluation protocols, which may push forwards more cutting edge research in TSGV. Besides, we also share our insights on several promising directions, including three typical tasks with new and practical settings based on TSGV.