Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding
Not on the map. Why?
Abstract (arXiv)
Spatio-temporal video grounding (STVG) requires models to identify when a referred event occurs and localize the target entity throughout that interval. Existing multimodal large language models typically serialize dense localization trajectories autoregressively, causing decoding latency to grow with tube length and allowing localization errors to propagate across time. We introduce Parallel Tube Decoding (PTD), a generative formulation that decomposes grounding into a temporal block followed by time-conditioned spatial blocks decoded simultaneously. This removes both token-level and trajectory-level dependencies, reducing the sequential decoding depth to a fixed 1 + 1 rounds, independent of tube length. To enable parallel spatial generation, we introduce Decoupled Block Attention, which preserves access to shared video-query context while eliminating cross-box dependencies, together with localization-aware policy optimization for temporal boundaries and spatial geometry. On VidSTG, PTD reduces Tube Completion Latency by 79x and increases spatial decoding throughput by 92x over standard autoregressive decoding, while also improving grounding accuracy. With a compact 4B backbone, our model performs favorably well on VidSTG and HC-STVG, and generalizes zero-shot to temporal grounding, grounded VideoQA, and referring video object tracking. Our results show parallel tube generation is an efficient and effective alternative to autoregressive localization in videos.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 65 of its 66 references that are in the map. First posted on arXiv 28 Aug 2026.
Builds on
65 of 66 references.
References · 65
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024GPT-4o System Cardcited by 4,588
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
38 more
- 2025Large Language Diffusion Modelscited by 912
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
- 2025SAM 3: Segment Anything with Conceptscited by 906
- 2017Dense-Captioning Events in Videoscited by 793
- 2024PaliGemma: A versatile 3B VLM for transfercited by 696
- 2017TALL: Temporal Activity Localization via Language Querycited by 584
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
- 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
- 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
- 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
- 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
- 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
- 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
- 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
- 2018Video Object Segmentation with Language Referring Expressionscited by 162
- 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
- 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 139
- 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
- 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 119
- 2020Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentencescited by 118
- 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 106
- 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
- 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 100
- 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 98
- 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
- 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89
- 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 85
- 2024VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Groundingcited by 85
- 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 85
- 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 84
- 2025Detect Anything via Next Point Predictioncited by 73
- 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65
- 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 59
- 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videoscited by 51
- 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
- 2024Context-Guided Spatio-Temporal Video Groundingcited by 46
- 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
- 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
50 most cited of 65 All 65 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding
- 2025OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex Scenarios
- 2021End-to-End Dense Video Grounding via Parallel Regression
- 2025STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding
- 2025TubeRMC: Tube-conditioned Reconstruction with Mutual Constraints for Weakly-supervised Spatio-Temporal Video Grounding
- 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding
- 2026GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
- 2026Agentic Spatio-Temporal Grounding via Collaborative Reasoning