# Video Activity Localisation with Uncertainties in Temporal Boundary

arXiv: https://arxiv.org/abs/2206.12923 · HTML: https://mlsanity.com/paper/2206.12923/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 26 Jun 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Dense Regression Network for Video Grounding](https://mlsanity.com/paper/2004.03545/) (2020)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Interventional Video Grounding with Dual Contrastive Learning](https://mlsanity.com/paper/2106.11013/) (2021)
- [Boundary Proposal Network for Two-Stage Natural Language Video Localization](https://mlsanity.com/paper/2103.08109/) (2021)
- [MAC: Mining Activity Concepts for Language-based Temporal Localization](https://mlsanity.com/paper/1811.08925/) (2018)
- [ExCL: Extractive Clip Localization Using Natural Language Descriptions](https://mlsanity.com/paper/1904.02755/) (2019)
- [Uncovering Hidden Challenges in Query-Based Video Moment Retrieval](https://mlsanity.com/paper/2009.00325/) (2020)
- [Dynamic Coattention Networks For Question Answering](https://mlsanity.com/paper/1611.01604/) (2016)
- [Cross-Sentence Temporal and Semantic Relations in Video Activity Localisation](https://mlsanity.com/paper/2107.11443/) (2021)
- [Exploiting Temporal Relationships in Video Moment Localization with Natural Language](https://mlsanity.com/paper/1908.03846/) (2019)
- [Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding](https://mlsanity.com/paper/2103.16848/) (2021)
- [Boundary Uncertainty in a Single-Stage Temporal Action Localization Network](https://mlsanity.com/paper/2008.11170/) (2020)

## Built on by (most cited first, 24 of 24)

- [Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training](https://mlsanity.com/paper/2303.00040/) (2023)
- [Training-free Video Temporal Grounding using Large-scale Pre-trained Models](https://mlsanity.com/paper/2408.16219/) (2024)
- [Zero-Shot Video Moment Retrieval from Frozen Vision-Language Models](https://mlsanity.com/paper/2309.00661/) (2023)
- [Background-aware Moment Detection for Video Moment Retrieval](https://mlsanity.com/paper/2306.02728/) (2023)
- [Enrich and Detect: Video Temporal Grounding with Multimodal LLMs](https://mlsanity.com/paper/2510.17023/) (2025)
- [Boundary-Denoising for Video Activity Localization](https://mlsanity.com/paper/2304.02934/) (2023)
- [MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval](https://mlsanity.com/paper/2406.17880/) (2024)
- [Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos](https://mlsanity.com/paper/2303.06378/) (2023)
- [TAG: A Simple Yet Effective Temporal-Aware Approach for Zero-Shot Video Temporal Grounding](https://mlsanity.com/paper/2508.07925/) (2025)
- [Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Grounding](https://mlsanity.com/paper/2508.04546/) (2025)
- [SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses](https://mlsanity.com/paper/2408.01669/) (2024)
- [Efficient Temporal Sentence Grounding in Videos with Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2308.03725/) (2023)
- [Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval](https://mlsanity.com/paper/2508.04273/) (2025)
- [DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding](https://mlsanity.com/paper/2607.00672/) (2026)
- [Video sentence grounding with temporally global textual knowledge](https://mlsanity.com/paper/2404.13611/) (2024)
- [Multi-Stage Boundary-Aware Transformer Network for Action Segmentation in Untrimmed Surgical Videos](https://mlsanity.com/paper/2504.18756/) (2025)
- [LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization](https://mlsanity.com/paper/2505.24282/) (2025)
- [OVG-HQ: Online Video Grounding with Hybrid-modal Queries](https://mlsanity.com/paper/2508.11903/) (2025)
- [Multi-Level LVLM Guidance for Untrimmed Video Action Recognition](https://mlsanity.com/paper/2508.17442/) (2025)
- [See, Rank, and Filter: Important Word-Aware Clip Filtering via Scene Understanding for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2511.22906/) (2025)
- [HERO: Hierarchical Embedding-Refinement for Open-Vocabulary Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2603.06732/) (2026)
- [Conformal Coverage Guarantees for Any Video Temporal Grounder](https://mlsanity.com/paper/2608.07434/) (2026)
- [ID-VTG: Image-Disambiguated Video Temporal Grounding](https://mlsanity.com/paper/2608.20127/) (2026)
- [DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding](https://mlsanity.com/paper/2609.08850/) (2026)

## Abstract (arXiv)

Current methods for video activity localisation over time assume implicitly that activity temporal boundaries labelled for model training are determined and precise. However, in unscripted natural videos, different activities mostly transit smoothly, so that it is intrinsically ambiguous to determine in labelling precisely when an activity starts and ends over time. Such uncertainties in temporal labelling are currently ignored in model training, resulting in learning mis-matched video-text correlation with poor generalisation in test. In this work, we solve this problem by introducing Elastic Moment Bounding (EMB) to accommodate flexible and adaptive activity temporal boundaries towards modelling universally interpretable video-text correlation with tolerance to underlying temporal uncertainties in pre-fixed annotations. Specifically, we construct elastic boundaries adaptively by mining and discovering frame-wise temporal endpoints that can maximise the alignment between video segments and query sentences. To enable both more accurate matching (segment content attention) and more robust localisation (segment elastic boundaries), we optimise the selection of frame-wise endpoints subject to segment-wise contents by a novel Guided Attention mechanism. Extensive experiments on three video activity localisation benchmarks demonstrate compellingly the EMB's advantages over existing methods without modelling uncertainty.