mlsanityData to 7 Oct 2026

MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence

arXiv 2505.23764 ↗First posted 29 May 2025cs.CV cs.CLcited by 169

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Spatial intelligence is essential for multimodal large language models (MLLMs) operating in the complex physical world. Existing benchmarks, however, probe only single-image relations and thus fail to assess the multi-image spatial reasoning that real-world deployments demand. We introduce MMSI-Bench, a VQA benchmark dedicated to multi-image spatial intelligence. Six 3D-vision researchers spent more than 300 hours meticulously crafting 1,000 challenging, unambiguous multiple-choice questions from over 120,000 images, each paired with carefully designed distractors and a stepwise reasoning process. We conduct extensive experiments and evaluate 37 open-source and proprietary MLLMs, observing a wide gap: the strongest open-source model attains roughly 30% accuracy and OpenAI's GPT-5 reasoning model reaches 40%, while humans score 97%. These results underscore the challenging nature of MMSI-Bench and the substantial headroom for future research. Leveraging the annotated reasoning processes, we also provide an automated error analysis pipeline that diagnoses four dominant failure modes, including (1) grounding errors, (2) overlap-matching and scene-reconstruction errors, (3) situation-transformation reasoning errors, and (4) spatial-logic errors, offering insights for advancing spatial intelligence. Project page: https://runsenxu.com/projects/MMSI_Bench .

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 71 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

47 of 71 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?25 Mar
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  3. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark10 Dec
  4. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models9 Nov
  5. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models15 Sep
  6. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  7. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  8. 2022Visual Spatial Reasoning30 Apr

Other references · 39

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
27 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  5. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  8. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  9. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  10. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  11. 2025Seed1.5-VL Technical Reportcited by 367
  12. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  13. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  14. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  15. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  16. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  17. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
  18. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  19. 2024Does Spatial Cognition Emerge in Frontier Models?cited by 65
  20. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  21. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  22. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  23. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Modelscited by 52
  24. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  25. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 36
  26. 2023PATS: Patch Area Transportation with Subdivision for Local Feature Matchingcited by 33
  27. 2024Towards Foundation Models for 3D Vision: How Close Are We?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 40; 2026: 129.

169 papers.

Sort:

By area:

  1. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  2. 2025Visual Spatial Tuningcited by 85
  3. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 75
  4. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  5. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  6. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 58
  7. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  8. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  9. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 41
  10. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 39
  11. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 34
  12. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
38 more
  1. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  2. 2026STEP3-VL-10B Technical Reportcited by 28
  3. 2025Geometrically-Constrained Agent for Spatial Reasoningcited by 25
  4. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
  5. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 22
  6. 2025DSI-Bench: A Benchmark for Dynamic Spatial Intelligencecited by 20
  7. 2025AdaTooler-V: Adaptive Tool-Use for Images and Videoscited by 20
  8. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 19
  9. 2026ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loopcited by 19
  10. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 18
  11. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 18
  12. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 18
  13. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  14. 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 16
  15. 2025Mull-Tokens: Modality-Agnostic Latent Thinkingcited by 15
  16. 2026Current Agents Fail to Leverage World Model as Tool for Foresightcited by 15
  17. 2026Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Modelscited by 14
  18. 2026Thinking with Geometry: Active Geometry Integration for Spatial Reasoningcited by 14
  19. 2026SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoningcited by 13
  20. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 12
  21. 2026When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoningcited by 12
  22. 2025SpatialTree: How Spatial Abilities Branch Out in MLLMscited by 11
  23. 2025STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligencecited by 10
  24. 2025Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Modelscited by 10
  25. 2025Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigationcited by 10
  26. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 10
  27. 2025Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Modelscited by 10
  28. 2026JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generationcited by 10
  29. 2025Can World Models Benefit VLMs for World Dynamics?cited by 9
  30. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  31. 2026CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environmentscited by 8
  32. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 8
  33. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoningcited by 8
  34. 2026Context Unrolling in Omni Modelscited by 8
  35. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 8
  36. 2025Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Modelscited by 7
  37. 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
  38. 2026S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligencecited by 7

50 most cited of 169 All 169 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  2. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence
  3. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
  4. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  5. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
  6. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks
  7. 2025InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models
  8. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks