# Can Multimodal Large Language Models Understand Spatial Relations?

arXiv: https://arxiv.org/abs/2505.19015 · HTML: https://mlsanity.com/paper/2505.19015/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 25 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models](https://mlsanity.com/paper/2310.16436/) (2023)
- [T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering](https://mlsanity.com/paper/2305.03453/) (2023)
- [CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs](https://mlsanity.com/paper/2401.02582/) (2024)
- [SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition](https://mlsanity.com/paper/1908.02660/) (2019)
- [A Survey for Foundation Models in Autonomous Driving](https://mlsanity.com/paper/2402.01105/) (2024)
- [Texts as Images in Prompt Tuning for Multi-Label Image Recognition](https://mlsanity.com/paper/2211.12739/) (2022)
- [Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3D](https://mlsanity.com/paper/2012.01634/) (2020)
- [Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning](https://mlsanity.com/paper/2310.00647/) (2023)
- [MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2309.04790/) (2023)
- [Can Transformers Capture Spatial Relations between Objects?](https://mlsanity.com/paper/2403.00729/) (2024)

## Built on by (most cited first, 20 of 20)

- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs](https://mlsanity.com/paper/2512.19683/) (2025)
- [Spatial-aware Vision Language Model for Autonomous Driving](https://mlsanity.com/paper/2512.24331/) (2025)
- [TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoning](https://mlsanity.com/paper/2601.16520/) (2026)
- [The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents](https://mlsanity.com/paper/2605.20544/) (2026)
- [MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model](https://mlsanity.com/paper/2603.18892/) (2026)
- [Are VLMs Lost Between Sky and Space? LinkS2Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligence](https://mlsanity.com/paper/2604.02020/) (2026)
- [Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics](https://mlsanity.com/paper/2506.06006/) (2025)
- [Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?](https://mlsanity.com/paper/2605.30557/) (2026)
- [MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook](https://mlsanity.com/paper/2509.14142/) (2025)
- [AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretation](https://mlsanity.com/paper/2601.03191/) (2026)
- [PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning](https://mlsanity.com/paper/2605.19382/) (2026)
- [BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling](https://mlsanity.com/paper/2606.20146/) (2026)
- [Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection](https://mlsanity.com/paper/2511.20162/) (2025)
- [On Semiotic-Grounded Interpretive Evaluation of Generative Art](https://mlsanity.com/paper/2604.08641/) (2026)
- [Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence](https://mlsanity.com/paper/2605.25334/) (2026)
- [TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs](https://mlsanity.com/paper/2608.05616/) (2026)
- [UniGeo: A Multi-modal Large Language Model for Text-Guided Cross-View Geo-Localization](https://mlsanity.com/paper/2608.26722/) (2026)
- [PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models](https://mlsanity.com/paper/2609.14973/) (2026)
- [SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models](https://mlsanity.com/paper/2609.38716/) (2026)

## Abstract (arXiv)

Spatial relation reasoning is a crucial task for multimodal large language models (MLLMs) to understand the objective world. However, current benchmarks have issues like relying on bounding boxes, ignoring perspective substitutions, or allowing questions to be answered using only the model's prior knowledge without image understanding. To address these issues, we introduce SpatialMQA, a human-annotated spatial relation reasoning benchmark based on COCO2017, which enables MLLMs to focus more on understanding images in the objective world. To ensure data quality, we design a well-tailored annotation procedure, resulting in SpatialMQA consisting of 5,392 samples. Based on this benchmark, a series of closed- and open-source MLLMs are implemented and the results indicate that the current state-of-the-art MLLM achieves only 48.14% accuracy, far below the human-level accuracy of 98.40%. Extensive experimental analyses are also conducted, suggesting the future research directions. The benchmark and codes are available at https://github.com/ziyan-xiaoyu/SpatialMQA.git.