# Knowing Where to Focus: Event-aware Transformer for Video Grounding

arXiv: https://arxiv.org/abs/2308.06947 · HTML: https://mlsanity.com/paper/2308.06947/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 91 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 30 of 71)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Dense Regression Network for Video Grounding](https://mlsanity.com/paper/2004.03545/) (2020)
- [Fast Convergence of DETR with Spatially Modulated Co-Attention](https://mlsanity.com/paper/2101.07448/) (2021)
- [Support-set bottlenecks for video-text representation learning](https://mlsanity.com/paper/2010.02824/) (2020)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression](https://mlsanity.com/paper/1804.07014/) (2018)
- [Rethinking Transformer-based Set Prediction for Object Detection](https://mlsanity.com/paper/2011.10881/) (2020)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment](https://mlsanity.com/paper/1812.00087/) (2018)
- [Learning the Best Pooling Strategy for Visual Semantic Embedding](https://mlsanity.com/paper/2011.04305/) (2020)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Multilevel Language and Vision Integration for Text-to-Clip Retrieval](https://mlsanity.com/paper/1804.05113/) (2018)
- [Efficient DETR: Improving End-to-End Object Detector with Dense Prior](https://mlsanity.com/paper/2104.01318/) (2021)

## Built on by (most cited first, 30 of 79)

- [Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding](https://mlsanity.com/paper/2503.13377/) (2025)
- [ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos](https://mlsanity.com/paper/2406.19392/) (2024)
- [R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding](https://mlsanity.com/paper/2404.00801/) (2024)
- [Correlation-Guided Query-Dependency Calibration for Video Temporal Grounding](https://mlsanity.com/paper/2311.08835/) (2023)
- [Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval](https://mlsanity.com/paper/2407.15051/) (2024)
- [BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2312.00083/) (2023)
- [Training-free Video Temporal Grounding using Large-scale Pre-trained Models](https://mlsanity.com/paper/2408.16219/) (2024)
- [Universal Video Temporal Grounding with Generative Multi-modal Large Language Models](https://mlsanity.com/paper/2506.18883/) (2025)
- [SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding](https://mlsanity.com/paper/2404.01174/) (2024)
- [Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2404.09263/) (2024)
- [FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Grounding](https://mlsanity.com/paper/2412.13441/) (2024)
- [Saliency-Guided DETR for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2410.01615/) (2024)
- [VTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPT](https://mlsanity.com/paper/2403.02076/) (2024)
- [Chrono: A Simple Blueprint for Representing Time in MLLMs](https://mlsanity.com/paper/2406.18113/) (2024)
- [Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motion](https://mlsanity.com/paper/2404.11375/) (2024)
- [RGNet: A Unified Clip Retrieval and Grounding Network for Long Videos](https://mlsanity.com/paper/2312.06729/) (2023)
- [TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs](https://mlsanity.com/paper/2509.18056/) (2025)
- [Enrich and Detect: Video Temporal Grounding with Multimodal LLMs](https://mlsanity.com/paper/2510.17023/) (2025)
- [Improving Visual Recognition with Hyperbolical Visual Hierarchy Mapping](https://mlsanity.com/paper/2404.00974/) (2024)
- [Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding](https://mlsanity.com/paper/2403.11463/) (2024)
- [Beyond Uncertainty: Evidential Deep Learning for Robust Video Temporal Grounding](https://mlsanity.com/paper/2408.16272/) (2024)
- [Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval](https://mlsanity.com/paper/2502.08544/) (2025)
- [MS-DETR: Towards Effective Video Moment Retrieval and Highlight Detection by Joint Motion-Semantic Learning](https://mlsanity.com/paper/2507.12062/) (2025)
- [Tempo-R0: A Video-MLLM for Temporal Video Grounding through Efficient Temporal Sensing Reinforcement Learning](https://mlsanity.com/paper/2507.04702/) (2025)
- [LD-DETR: Loop Decoder DEtection TRansformer for Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2501.10787/) (2025)
- [Prototypes are Balanced Units for Efficient and Effective Partially Relevant Video Retrieval](https://mlsanity.com/paper/2504.13035/) (2025)
- [TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2512.03963/) (2025)
- [Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understanding](https://mlsanity.com/paper/2410.13598/) (2024)
- [TAG: A Simple Yet Effective Temporal-Aware Approach for Zero-Shot Video Temporal Grounding](https://mlsanity.com/paper/2508.07925/) (2025)
- [VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotations](https://mlsanity.com/paper/2510.23397/) (2025)

## Abstract (arXiv)

Recent DETR-based video grounding models have made the model directly predict moment timestamps without any hand-crafted components, such as a pre-defined proposal or non-maximum suppression, by learning moment queries. However, their input-agnostic moment queries inevitably overlook an intrinsic temporal structure of a video, providing limited positional information. In this paper, we formulate an event-aware dynamic moment query to enable the model to take the input-specific content and positional information of the video into account. To this end, we present two levels of reasoning: 1) Event reasoning that captures distinctive event units constituting a given video using a slot attention mechanism; and 2) moment reasoning that fuses the moment queries with a given sentence through a gated fusion transformer layer and learns interactions between the moment queries and video-sentence representations to predict moment timestamps. Extensive experiments demonstrate the effectiveness and efficiency of the event-aware dynamic moment queries, outperforming state-of-the-art approaches on several video grounding benchmarks.