# Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification

arXiv: https://arxiv.org/abs/2606.29023 · HTML: https://mlsanity.com/paper/2606.29023/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 40 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Builds on (most cited first, 30 of 34)

- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2504.06958/) (2025)
- [LaSOT: A High-quality Large-scale Single Object Tracking Benchmark](https://mlsanity.com/paper/2009.03465/) (2020)
- [SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes](https://mlsanity.com/paper/2304.05170/) (2023)
- [Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences](https://mlsanity.com/paper/2001.06891/) (2020)
- [Actor and Action Video Segmentation from a Sentence](https://mlsanity.com/paper/1803.07485/) (2018)
- [TubeDETR: Spatio-Temporal Video Grounding with Transformers](https://mlsanity.com/paper/2203.16434/) (2022)
- [V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning](https://mlsanity.com/paper/2503.11495/) (2025)
- [LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding](https://mlsanity.com/paper/2501.08282/) (2025)
- [Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding](https://mlsanity.com/paper/2604.05015/) (2026)
- [VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos](https://mlsanity.com/paper/2411.04923/) (2024)
- [Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence](https://mlsanity.com/paper/2510.20579/) (2025)
- [Elysium: Exploring Object-level Perception in Videos via MLLM](https://mlsanity.com/paper/2403.16558/) (2024)
- [Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2209.13306/) (2022)
- [Object Referring in Videos with Language and Human Gaze](https://mlsanity.com/paper/1801.01582/) (2018)
- [GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents](https://mlsanity.com/paper/2604.26752/) (2026)
- [Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2401.00901/) (2023)
- [ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2411.15268/) (2024)
- [SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capability](https://mlsanity.com/paper/2503.13983/) (2025)
- [Spatio-temporal Person Retrieval via Natural Language Queries](https://mlsanity.com/paper/1704.07945/) (2017)
- [Object-Aware Multi-Branch Relation Networks for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2008.06941/) (2020)
- [Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2502.11168/) (2025)
- [Augmented 2D-TAN: A Two-stage Approach for Human-centric Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2106.10634/) (2021)
- [Enrich and Detect: Video Temporal Grounding with Multimodal LLMs](https://mlsanity.com/paper/2510.17023/) (2025)
- [Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2511.21375/) (2025)
- [ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking](https://mlsanity.com/paper/2504.09195/) (2025)
- [Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2509.15178/) (2025)
- [VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification](https://mlsanity.com/paper/2604.01569/) (2026)

## Abstract (arXiv)

Spatio-temporal grounding in long videos requires precise temporal localization and robust object tracking conditioned on natural-language queries. While recent vision-language models (VLMs) show strong reasoning ability, directly applying frame-by-frame inference to long sequences is computationally expensive and unstable. We propose a practical pipeline that shifts from frame-level to second-level tracking and performs cross-second smoothing to preserve continuity while reducing sequence length. To improve reasoning supervision, we synthesize chain-of-thought style trajectories using advanced multimodal models for temporal localization and target selection, and replace generated spatio-temporal coordinates with ground-truth annotations to avoid noisy supervision. We further optimize the policy with reinforcement learning using a verifier based on tIoU+mvIoU. Experiments across multiple FPS settings show that our method achieves a strong trade-off between efficiency and localization quality.