# Detector-Empowered Video Large Language Model for Efficient Spatio-Temporal Grounding

arXiv: https://arxiv.org/abs/2512.06673 · HTML: https://mlsanity.com/paper/2512.06673/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 73 references that are in the map.
- First posted on arXiv 7 Dec 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 41)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling](https://mlsanity.com/paper/2111.12681/) (2021)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [LITA: Language Instructed Temporal-Localization Assistant](https://mlsanity.com/paper/2403.19046/) (2024)
- [Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning](https://mlsanity.com/paper/2402.11435/) (2024)
- [AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning](https://mlsanity.com/paper/2103.16002/) (2021)
- [Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences](https://mlsanity.com/paper/2001.06891/) (2020)
- [One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos](https://mlsanity.com/paper/2409.19603/) (2024)
- [TRACE: Temporal Grounding Video LLM via Causal Event Modeling](https://mlsanity.com/paper/2410.05643/) (2024)
- [HawkEye: Training Video-Text LLMs for Grounding Text in Videos](https://mlsanity.com/paper/2403.10228/) (2024)
- [Human-centric Spatio-Temporal Video Grounding With Visual Transformers](https://mlsanity.com/paper/2011.05049/) (2020)
- [TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability](https://mlsanity.com/paper/2411.18211/) (2024)
- [An Open and Comprehensive Pipeline for Unified Object Grounding and Detection](https://mlsanity.com/paper/2401.02361/) (2024)
- [LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding](https://mlsanity.com/paper/2501.08282/) (2025)

## Built on by (most cited first, 1 of 1)

- [Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2608.28192/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) are rapidly expanding from general video understanding to finer-grained understanding such as spatio-temporal video grounding (STVG) and reasoning. In these tasks, an MLLM must localize the user-queried target in time and space and take the results as evidence for reasoning. Existing MLLM methods mainly follow two paradigms: (1) Direct Localization, which outputs STVG results with extra alignment modules or specialized decoders; and (2) Candidate-based Selection, which first constructs tube-level candidates and then selects the relevant one by an MLLM. However, both suffer from a serious efficiency bottleneck: the former incurs linearly growing decoding cost as the queried temporal span increases, while the latter relies on costly candidate construction. To break this bottleneck, we propose DEViL, a detector-empowered Video-LLM with a simple key idea: offloading dense spatial grounding from the MLLM to a fully parallelizable, well-trained detector. Specifically, DEViL distills the query into a detector-compatible reference-semantic token, which replaces the detector's text embedding to enable spatial grounding in a single pass. Then, we design temporal consistency regularization to match objects across frames and enforce their coherence over time. In this way, DEViL avoids long coordinate decoding and heavy candidate pipelines. Extensive experiments show that DEViL achieves strong performance (43.1% m_vIoU on HC-STVG) with superior efficiency (14.33 FPS), while preserving the general reasoning capacity of the MLLM backbone.