mlsanityData to 7 Oct 2026

V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning

arXiv 2503.11495 ↗First posted 14 Mar 2025cs.CVcited by 66

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Human processes video reasoning in a sequential spatio-temporal reasoning logic, we first identify the relevant frames ("when") and then analyse the spatial relationships ("where") between key objects, and finally leverage these relationships to draw inferences ("what"). However, can Video Large Language Models (Video-LLMs) also "reason through a sequential spatio-temporal logic" in videos? Existing Video-LLM benchmarks primarily focus on assessing object presence, neglecting relational reasoning. Consequently, it is difficult to measure whether a model truly comprehends object interactions (actions/events) in videos or merely relies on pre-trained "memory" of co-occurrences as biases in generating answers. In this work, we introduce a Video Spatio-Temporal Reasoning (V-STaR) benchmark to address these shortcomings. The key idea is to decompose video understanding into a Reverse Spatio-Temporal Reasoning (RSTR) task that simultaneously evaluates what objects are present, when events occur, and where they are located while capturing the underlying Chain-of-thought (CoT) logic. To support this evaluation, we construct a dataset to elicit the spatial-temporal reasoning process of Video-LLMs. It contains coarse-to-fine CoT questions generated by a semi-automated GPT-4-powered pipeline, embedding explicit reasoning chains to mimic human cognition. Experiments from 14 Video-LLMs on our V-STaR reveal significant gaps between current Video-LLMs and the needs for robust and consistent spatio-temporal reasoning.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 46 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

43 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024VISA: Reasoning Video Object Segmentation via Large Language Models16 Jul
  3. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos27 Jun
  4. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models8 Sep
  5. 2022From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering30 May

Other references · 38

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  7. 2016Modeling Context in Referring Expressionscited by 1,054
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  9. 2017Dense-Captioning Events in Videoscited by 789
  10. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  11. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  12. 2017TALL: Temporal Activity Localization via Language Querycited by 581
26 more
  1. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  2. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  4. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  5. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  6. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  7. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  8. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 283
  9. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  10. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  11. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 234
  12. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  13. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  14. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  15. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  16. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  17. 2020Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentencescited by 116
  18. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  19. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  20. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  21. 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 83
  22. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  23. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  24. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  25. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 50
  26. 2024Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 24; 2026: 42.

66 papers.

Sort:

By area:

  1. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
  2. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  3. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  4. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  5. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  6. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  7. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1cited by 24
  8. 2026Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representationcited by 21
  9. 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologiescited by 17
  10. 2025MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Groundingcited by 11
  11. 2025Video-EM: Event-Centric Episodic Memory for Long-Form Video Understandingcited by 10
  12. 2025STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligencecited by 10
38 more
  1. 2026Vero: An Open RL Recipe for General Visual Reasoningcited by 10
  2. 2026VISD: Enhancing Video Reasoning via Structured Self-Distillationcited by 9
  3. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  4. 2025Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Modelscited by 7
  5. 2026STReasoner: Empowering LLMs for Spatio-Temporal Reasoning in Time Series via Spatial-Aware Reinforcement Learningcited by 7
  6. 2025ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videoscited by 6
  7. 2026LanteRn: Latent Visual Structured Reasoningcited by 6
  8. 2026VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verificationcited by 6
  9. 2025TimeScope: Towards Task-Oriented Temporal Grounding In Long Videoscited by 5
  10. 2025Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graphcited by 5
  11. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  12. 2025Benchmarking Spatiotemporal Reasoning in LLMs and Reasoning Models: Capabilities and Challengescited by 4
  13. 2026SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environmentscited by 4
  14. 2026Leveraging Latent Visual Reasoning in Silencecited by 4
  15. 2026Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoningcited by 4
  16. 2026Cut to the Chase: Training-free Multimodal Summarization via Chain-of-Eventscited by 3
  17. 2026VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Promptingcited by 3
  18. 2026ViKey: Enhancing Temporal Understanding in Videos via Visual Promptingcited by 3
  19. 2026Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Modelscited by 3
  20. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  21. 2025Video Reasoning without Trainingcited by 2
  22. 2025SPHINX: A Synthetic Environment for Visual Perception and Reasoningcited by 2
  23. 2025Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspectivecited by 2
  24. 2025GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Modelscited by 2
  25. 2026Can LLMs Perceive Time? An Empirical Investigationcited by 2
  26. 2025R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarioscited by 1
  27. 2025M3-Verse: A "Spot the Difference" Challenge for Large Multimodal Modelscited by 1
  28. 2026Weaver: End-to-End Agentic System Training for Video Interleaved Reasoningcited by 1
  29. 2026Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidencecited by 1
  30. 2026HumanOmni-Speaker: Identifying Who said What and Whencited by 1
  31. 2026Grounding Video Reasoning in Physical Signalscited by 1
  32. 2026STAR: Failure-Aware Markovian Routing for Multi-Agent Spatiotemporal Reasoningcited by 1
  33. 2026Maestro: Reinforcement Learning to Orchestrate Hierarchical Model-Skill Ensemblescited by 1
  34. 2026Evidence-Backed Video Question Answeringcited by 1
  35. 2025INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Modelscited by 0
  36. 2026Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understandingcited by 0
  37. 2026PinpointQA: A Benchmark for Small Object-Centric Spatial Understanding in Indoor Videoscited by 0
  38. 2026Lost in Motion: Vision Language Models Fail the Dynamic Gauges Testcited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning
  2. 2026VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis
  3. 2026CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering
  4. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs
  5. 2026STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models
  6. 2024STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training
  7. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMs
  8. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task