mlsanityData to 14 Oct 2026

STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?

arXiv 2503.23765 ↗First posted 31 Mar 2025cs.CVcited by 79

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The use of Multimodal Large Language Models (MLLMs) as an end-to-end solution for Embodied AI and Autonomous Driving has become a prevailing trend. While MLLMs have been extensively studied for visual semantic understanding tasks, their ability to perform precise and quantitative spatial-temporal understanding in real-world applications remains largely unexamined, leading to uncertain prospects. To evaluate models' Spatial-Temporal Intelligence, we introduce STI-Bench, a benchmark designed to evaluate MLLMs' spatial-temporal understanding through challenging tasks such as estimating and predicting the appearance, pose, displacement, and motion of objects. Our benchmark encompasses a wide range of robot and vehicle operations across desktop, indoor, and outdoor scenarios. The extensive experiments reveals that the state-of-the-art MLLMs still struggle in real-world spatial-temporal understanding, especially in tasks requiring precise distance estimation and motion analysis.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 51 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

37 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents13 Feb
  2. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents21 Jan
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  4. 2024SpatialBot: Precise Spatial Understanding with Vision Language Models19 Jun
  5. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models9 Jun
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 30

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
18 more
  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  4. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  5. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  6. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  7. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  8. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  9. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  10. 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 236
  11. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  12. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  13. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  14. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  15. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 148
  16. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 86
  17. 2024Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Trackingcited by 50
  18. 2023WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Modelscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 35; 2026: 44.

79 papers.

Sort:

By area:

  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  2. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 159
  3. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 118
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  5. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 90
  6. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  7. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 41
  8. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  9. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 38
  10. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  11. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  12. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
38 more
  1. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 24
  2. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 21
  3. 2026VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twicecited by 21
  4. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 19
  5. 2025I2-NeRF: Learning Neural Radiance Fields Under Physically-Grounded Media Interactionscited by 17
  6. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Modelcited by 15
  7. 2026SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoningcited by 15
  8. 2025VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimizationcited by 12
  9. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 12
  10. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D Worldcited by 12
  11. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  12. 2026Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Trainingcited by 11
  13. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  14. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 10
  15. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  16. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suitecited by 9
  17. 2025Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regressioncited by 8
  18. 2026Make Geometry Matter for Spatial Reasoningcited by 8
  19. 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoningcited by 7
  20. 2025Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graphcited by 6
  21. 2026OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMscited by 6
  22. 2025HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Drivingcited by 4
  23. 2026ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Modelscited by 4
  24. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  25. 2026Towards Spatial Supersensing in the Wildcited by 4
  26. 2026Self-Evolving Neuro-Symbolic Skills for Tool-Augmented Spatial Reasoningcited by 4
  27. 2025MIRAGE: A Multi-modal Benchmark for Spatial Perception, Reasoning, and Intelligencecited by 3
  28. 2025MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agentscited by 3
  29. 2025Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmarkcited by 3
  30. 2026Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spacescited by 3
  31. 2026Are VLMs Lost Between Sky and Space? LinkS2Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligencecited by 3
  32. 2026LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Modelscited by 3
  33. 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
  34. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Modelscited by 2
  35. 20254D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillationcited by 2
  36. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  37. 2026SSR: Pushing the Limit of Spatial Intelligence with Structured Scene Reasoningcited by 2
  38. 2026Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoningcited by 2

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?
  2. 2025OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding
  3. 2026Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models
  4. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
  5. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture
  6. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models
  7. 2025ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models
  8. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks