# Context-aware Biaffine Localizing Network for Temporal Sentence Grounding

arXiv: https://arxiv.org/abs/2103.11555 · HTML: https://mlsanity.com/paper/2103.11555/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 22 Mar 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Dense Regression Network for Video Grounding](https://mlsanity.com/paper/2004.03545/) (2020)
- [To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression](https://mlsanity.com/paper/1804.07014/) (2018)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment](https://mlsanity.com/paper/1812.00087/) (2018)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Multilevel Language and Vision Integration for Text-to-Clip Retrieval](https://mlsanity.com/paper/1804.05113/) (2018)
- [Weakly Supervised Video Moment Retrieval From Text Queries](https://mlsanity.com/paper/1904.03282/) (2019)
- [MAC: Mining Activity Concepts for Language-based Temporal Localization](https://mlsanity.com/paper/1811.08925/) (2018)
- [Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction](https://mlsanity.com/paper/1909.05010/) (2019)
- [Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention](https://mlsanity.com/paper/1908.07236/) (2019)
- [Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos](https://mlsanity.com/paper/1906.02497/) (2019)
- [Jointly Cross- and Self-Modal Graph Attention Network for Query-Based Moment Localization](https://mlsanity.com/paper/2008.01403/) (2020)
- [Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos](https://mlsanity.com/paper/1901.06829/) (2019)
- [Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos](https://mlsanity.com/paper/2007.14164/) (2020)

## Built on by (most cited first, 30 of 107)

- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding](https://mlsanity.com/paper/2109.04872/) (2021)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [MomentDiff: Generative Video Moment Retrieval from Random to Real](https://mlsanity.com/paper/2307.02869/) (2023)
- [On Pursuit of Designing Multi-modal Transformer for Video Grounding](https://mlsanity.com/paper/2109.06085/) (2021)
- [Harmonizing Visual Text Comprehension and Generation](https://mlsanity.com/paper/2407.16364/) (2024)
- [Memory-Guided Semantic Learning Network for Temporal Sentence Grounding](https://mlsanity.com/paper/2201.00454/) (2022)
- [Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval](https://mlsanity.com/paper/2209.11572/) (2022)
- [Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos](https://mlsanity.com/paper/2109.06398/) (2021)
- [You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos](https://mlsanity.com/paper/2303.07863/) (2023)
- [Hierarchical Local-Global Transformer for Temporal Sentence Grounding](https://mlsanity.com/paper/2208.14882/) (2022)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [SnAG: Scalable and Accurate Video Grounding](https://mlsanity.com/paper/2404.02257/) (2024)
- [Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding](https://mlsanity.com/paper/2203.02966/) (2022)
- [Reducing the Vision and Language Bias for Temporal Sentence Grounding](https://mlsanity.com/paper/2207.13457/) (2022)
- [Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training](https://mlsanity.com/paper/2303.00040/) (2023)
- [Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding](https://mlsanity.com/paper/2109.06400/) (2021)
- [Correlation-Guided Query-Dependency Calibration for Video Temporal Grounding](https://mlsanity.com/paper/2311.08835/) (2023)
- [Unsupervised Temporal Video Grounding with Deep Semantic Clustering](https://mlsanity.com/paper/2201.05307/) (2022)
- [Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval](https://mlsanity.com/paper/2407.15051/) (2024)
- [A Survey on Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2109.08039/) (2021)
- [BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2312.00083/) (2023)
- [Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval](https://mlsanity.com/paper/2312.12155/) (2023)
- [Hypotheses Tree Building for One-Shot Temporal Sentence Localization](https://mlsanity.com/paper/2301.01871/) (2023)
- [Exploring Motion and Appearance Information for Temporal Sentence Grounding](https://mlsanity.com/paper/2201.00457/) (2022)
- [FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Grounding](https://mlsanity.com/paper/2412.13441/) (2024)
- [EgoEnv: Human-centric environment representations from egocentric video](https://mlsanity.com/paper/2207.11365/) (2022)
- [Can Shuffling Video Benefit Temporal Bias Problem: A Novel Training Framework for Temporal Grounding](https://mlsanity.com/paper/2207.14698/) (2022)
- [Language-free Training for Zero-shot Video Grounding](https://mlsanity.com/paper/2210.12977/) (2022)
- [ViGT: Proposal-free Video Grounding with Learnable Token in Transformer](https://mlsanity.com/paper/2308.06009/) (2023)

## Abstract (arXiv)

This paper addresses the problem of temporal sentence grounding (TSG), which aims to identify the temporal boundary of a specific segment from an untrimmed video by a sentence query. Previous works either compare pre-defined candidate segments with the query and select the best one by ranking, or directly regress the boundary timestamps of the target segment. In this paper, we propose a novel localization framework that scores all pairs of start and end indices within the video simultaneously with a biaffine mechanism. In particular, we present a Context-aware Biaffine Localizing Network (CBLN) which incorporates both local and global contexts into features of each start/end position for biaffine-based localization. The local contexts from the adjacent frames help distinguish the visually similar appearance, and the global contexts from the entire video contribute to reasoning the temporal relation. Besides, we also develop a multi-modal self-attention module to provide fine-grained query-guided video representation for this biaffine strategy. Extensive experiments show that our CBLN significantly outperforms state-of-the-arts on three public datasets (ActivityNet Captions, TACoS, and Charades-STA), demonstrating the effectiveness of the proposed localization framework.