# An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models

arXiv: https://arxiv.org/abs/2411.06048 · HTML: https://mlsanity.com/paper/2411.06048/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 9 Nov 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [KOSMOS-2.5: A Multimodal Literate Model](https://mlsanity.com/paper/2309.11419/) (2023)
- [Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models](https://mlsanity.com/paper/2402.12058/) (2024)
- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs](https://mlsanity.com/paper/2402.11199/) (2024)

## Built on by (most cited first, 30 of 50)

- [MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence](https://mlsanity.com/paper/2505.23764/) (2025)
- [RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics](https://mlsanity.com/paper/2411.16537/) (2024)
- [ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models](https://mlsanity.com/paper/2505.21500/) (2025)
- [OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models](https://mlsanity.com/paper/2506.03135/) (2025)
- [Baichuan-Omni-1.5 Technical Report](https://mlsanity.com/paper/2501.15368/) (2025)
- [MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs](https://mlsanity.com/paper/2503.13111/) (2025)
- [Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models](https://mlsanity.com/paper/2505.17015/) (2025)
- [LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?](https://mlsanity.com/paper/2503.19990/) (2025)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Space](https://mlsanity.com/paper/2503.11094/) (2025)
- [SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models](https://mlsanity.com/paper/2509.17664/) (2025)
- [ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps](https://mlsanity.com/paper/2505.18675/) (2025)
- [Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes](https://mlsanity.com/paper/2510.19400/) (2025)
- [Current Agents Fail to Leverage World Model as Tool for Foresight](https://mlsanity.com/paper/2601.03905/) (2026)
- [Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs](https://mlsanity.com/paper/2506.22146/) (2025)
- [From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs](https://mlsanity.com/paper/2512.19683/) (2025)
- [On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs](https://mlsanity.com/paper/2602.12506/) (2026)
- [Can World Models Benefit VLMs for World Dynamics?](https://mlsanity.com/paper/2510.00855/) (2025)
- [Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability](https://mlsanity.com/paper/2506.02073/) (2025)
- [Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT](https://mlsanity.com/paper/2505.24182/) (2025)
- [LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Tasks](https://mlsanity.com/paper/2507.20174/) (2025)
- [Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fields](https://mlsanity.com/paper/2602.00148/) (2026)
- [SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models](https://mlsanity.com/paper/2602.20901/) (2026)
- [ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models](https://mlsanity.com/paper/2505.13180/) (2025)
- [Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models](https://mlsanity.com/paper/2505.05626/) (2025)
- [Caption This, Reason That: VLMs Caught in the Middle](https://mlsanity.com/paper/2505.21538/) (2025)
- [Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agent](https://mlsanity.com/paper/2508.11286/) (2025)
- [RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotation](https://mlsanity.com/paper/2508.13968/) (2025)
- [PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Models](https://mlsanity.com/paper/2512.19350/) (2025)
- [HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models](https://mlsanity.com/paper/2603.25411/) (2026)

## Abstract (arXiv)

Large Multimodal Models (LMMs) have achieved strong performance across a range of vision and language tasks. However, their spatial reasoning capabilities are under-investigated. In this paper, we construct a novel VQA dataset, Spatial-MM, to comprehensively study LMMs' spatial understanding and reasoning capabilities. Our analyses on object-relationship and multi-hop reasoning reveal several important findings. Firstly, bounding boxes and scene graphs, even synthetic ones, can significantly enhance LMMs' spatial reasoning. Secondly, LMMs struggle more with questions posed from the human perspective than the camera perspective about the image. Thirdly, chain of thought (CoT) prompting does not improve model performance on complex multi-hop questions involving spatial relations. % Moreover, spatial reasoning steps are much less accurate than non-spatial ones across MLLMs. Lastly, our perturbation analysis on GQA-spatial reveals that LMMs are much stronger at basic object detection than complex spatial reasoning. We believe our benchmark dataset and in-depth analyses can spark further research on LMMs spatial reasoning. Spatial-MM benchmark is available at: https://github.com/FatemehShiri/Spatial-MM