# Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning

arXiv: https://arxiv.org/abs/2511.21375 · HTML: https://mlsanity.com/paper/2511.21375/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Seed1.5-VL Technical Report](https://mlsanity.com/paper/2505.07062/) (2025)
- [Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding](https://mlsanity.com/paper/2503.13377/) (2025)
- [Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning](https://mlsanity.com/paper/2508.04416/) (2025)
- [UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning](https://mlsanity.com/paper/2505.14231/) (2025)
- [Flash-VStream: Efficient Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2506.23825/) (2025)
- [VGR: Visual Grounded Reasoning](https://mlsanity.com/paper/2506.11991/) (2025)
- [VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception](https://mlsanity.com/paper/2509.21100/) (2025)
- [The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer](https://mlsanity.com/paper/2504.10462/) (2025)
- [SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capability](https://mlsanity.com/paper/2503.13983/) (2025)
- [Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2502.11168/) (2025)
- [World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering](https://mlsanity.com/paper/2409.20424/) (2024)
- [Vidi: Large Multimodal Models for Video Understanding and Editing](https://mlsanity.com/paper/2504.15681/) (2025)
- [TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs](https://mlsanity.com/paper/2509.18056/) (2025)
- [Hierarchical Memory for Long Video QA](https://mlsanity.com/paper/2407.00603/) (2024)
- [OmniSTVG: Toward Spatio-Temporal Omni-Object Video Grounding](https://mlsanity.com/paper/2503.10500/) (2025)

## Built on by (most cited first, 12 of 12)

- [GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2602.14201/) (2026)
- [VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting](https://mlsanity.com/paper/2603.14659/) (2026)
- [Motion-o: Trajectory-Grounded Video Reasoning](https://mlsanity.com/paper/2603.18856/) (2026)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation](https://mlsanity.com/paper/2512.20117/) (2025)
- [Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding](https://mlsanity.com/paper/2604.08014/) (2026)
- [SER: Learning to Ground Video Reasoning with Semantic Evidence Rewards](https://mlsanity.com/paper/2606.24726/) (2026)
- [Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification](https://mlsanity.com/paper/2606.29023/) (2026)
- [AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models](https://mlsanity.com/paper/2607.02269/) (2026)
- [ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2607.13421/) (2026)
- [Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2608.28192/) (2026)
- [Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidence](https://mlsanity.com/paper/2610.17656/) (2026)

## Abstract (arXiv)

Spatio-temporal video grounding (STVG) requires localizing a target object in untrimmed videos both temporally and spatially from natural language descriptions. Despite their strong language understanding, multimodal large language models (MLLMs) underperform on STVG due to misaligned training objectives and weak fine-grained region-word alignment in standard visual encoders. To address this, we propose STVG-o1, the first framework that enables off-the-shelf MLLMs to achieve state-of-the-art STVG performance without any architectural modifications. Our method introduces a bounding-box chain-of-thought mechanism that explicitly reasons about spatio-temporal locations in an intermediate step before producing the final prediction. We further design a multi-dimensional reinforcement reward function consisting of format, consistency, temporal, spatial, and think rewards, which provides geometry-aware supervision through reinforcement fine-tuning. Evaluated on HCSTVG-v1/v2 and VidSTG, STVG-o1 sets new state-of-the-art results on HCSTVG, outperforming the best task-specific method by 7.3% mtIoU on HCSTVG-v1, matching specialized models on VidSTG, and surpassing all existing MLLM-based approaches by large margins. It also demonstrates strong open-vocabulary generalization across datasets, establishing MLLMs as viable and powerful backbones for precise spatio-temporal grounding. Our code and models will be released.