mlsanityData to 7 Oct 2026

iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs

arXiv 2502.03214 ↗First posted 5 Feb 2025cs.CL cs.AI cs.CVcited by 24

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) are known to struggle with spatial reasoning and visual alignment. To help overcome these limitations, we introduce iVISPAR, an interactive multimodal benchmark designed to evaluate the spatial reasoning capabilities of VLMs acting as agents. mboxiVISPAR is based on a variant of the sliding tile puzzle, a classic problem that demands logical planning, spatial awareness, and multi-step reasoning. The benchmark supports visual 3D, 2D, and text-based input modalities, enabling comprehensive assessments of VLMs' planning and reasoning skills. We evaluate a broad suite of state-of-the-art open-source and closed-source VLMs, comparing their performance while also providing optimal path solutions and a human baseline to assess the task's complexity and feasibility for humans. Results indicate that while VLMs perform better on 2D tasks compared to 3D or text-based settings, they struggle with complex spatial configurations and consistently fall short of human performance, illustrating the persistent challenge of visual alignment. This underscores critical gaps in current VLM capabilities, highlighting their limitations in achieving human-level cognition. Project website: https://microcosm.ai/ivispar

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 57 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

42 of 57 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment30 Oct
  2. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning21 Oct
  3. 2024ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities4 Oct
  4. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs2 Jul
  5. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models21 Jun
  6. 2024GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs19 Jun
  7. 2024SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models7 Jun
  8. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  9. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
  10. 2023Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning5 Oct
  11. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Models18 Aug
  12. 2022Visual Spatial Reasoning30 Apr

Other references · 30

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2024An Introduction to Vision-Language Modelingcited by 175
  11. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  12. 2022Linearly Mapping from Image to Text Spacecited by 139
18 more
  1. 2023Large Language Models for Robotics: A Surveycited by 132
  2. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  3. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 90
  4. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  5. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 71
  6. 2023Evaluating Spatial Understanding of Large Language Modelscited by 65
  7. 2016Learning Language Games through Interactioncited by 43
  8. 2017ShapeWorld - A new test methodology for multimodal language understandingcited by 37
  9. 2023GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language modelscited by 28
  10. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Taskscited by 23
  11. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
  12. 2024Exploring and Benchmarking the Planning Capabilities of Large Language Modelscited by 19
  13. 20233D-Aware Visual Question Answering about Parts, Poses and Occlusionscited by 16
  14. 2024PUZZLES: A Benchmark for Neural Algorithmic Reasoningcited by 12
  15. 2020Probing Contextual Language Models for Common Ground with Visual Representationscited by 9
  16. 2025Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumblecited by 9
  17. 2024Smart Vision-Language Reasonerscited by 5
  18. 2024SimBench: A Framework for Evaluating and Diagnosing LLM-Based Digital-Twin Generation for Multi-Physics Simulationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 13.

24 papers.

Sort:

By area:

  1. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  2. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  3. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 24
  4. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  5. 2025STAR-R1: Spatial TrAnsformation Reasoning by Reinforcing Multimodal LLMscited by 21
  6. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 13
  7. 2026SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environmentscited by 4
  8. 2025MIRAGE: A Multi-modal Benchmark for Spatial Perception, Reasoning, and Intelligencecited by 3
  9. 2025Pixels to Principles: Probing Intuitive Physics Understanding in Multimodal Language Modelscited by 2
  10. 2026Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Familiescited by 1
  11. 2026Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games?cited by 1
  12. 2026PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoningcited by 1
12 more
  1. 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
  2. 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Modelscited by 0
  3. 2025Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMscited by 0
  4. 2025A Multi-Modal Neuro-Symbolic Approach for Spatial Reasoning-Based Visual Grounding in Roboticscited by 0
  5. 2026Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structurescited by 0
  6. 2026Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoningcited by 0
  7. 2026SleepWalk: A Three-Tier Benchmark for Stress-Testing Instruction-Guided Vision-Language Navigationcited by 0
  8. 2026Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?cited by 0
  9. 2026SCALE: Self-Supervised Constraint-Aware Layout GEneration for Local P&R DRV Fixing at Advanced Nodescited by 0
  10. 2026ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Makingcited by 0
  11. 2026PolyComp: A Polycube-based Benchmark for Compositional 3D Spatial Reasoning in Multimodal Modelscited by 0
  12. 2026Investigating Human--AI Discrepancies via Multiple-Solution Problemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective
  2. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks
  3. 2025InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models
  4. 2026Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds
  5. 2025Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspective
  6. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  7. 2026WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning
  8. 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models