# Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models

arXiv: https://arxiv.org/abs/2502.08636 · HTML: https://mlsanity.com/paper/2502.08636/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 10 of its 53 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 10 of 10)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning](https://mlsanity.com/paper/2208.05358/) (2022)
- [SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models](https://mlsanity.com/paper/2412.07755/) (2024)
- [PromptCap: Prompt-Guided Task-Aware Image Captioning](https://mlsanity.com/paper/2211.09699/) (2022)

## Built on by (most cited first, 30 of 38)

- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2510.27606/) (2025)
- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [Holistic Evaluation of Multimodal LLMs on Spatial Intelligence](https://mlsanity.com/paper/2508.13142/) (2025)
- [SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence](https://mlsanity.com/paper/2505.17012/) (2025)
- [Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods](https://mlsanity.com/paper/2511.15722/) (2025)
- [SpaceVista: All-Scale Visual Spatial Reasoning from mm to km](https://mlsanity.com/paper/2510.09606/) (2025)
- [From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs](https://mlsanity.com/paper/2512.19683/) (2025)
- [FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks](https://mlsanity.com/paper/2502.17775/) (2025)
- [Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Models](https://mlsanity.com/paper/2508.04059/) (2025)
- [CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning](https://mlsanity.com/paper/2601.13304/) (2026)
- [Thinking with Spatial Code for Physical-World Video Reasoning](https://mlsanity.com/paper/2603.05591/) (2026)
- [Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents](https://mlsanity.com/paper/2605.08747/) (2026)
- [SceneActBench: Can Agents Act on the 3D Scenes They See?](https://mlsanity.com/paper/2607.22393/) (2026)
- [SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environments](https://mlsanity.com/paper/2604.14144/) (2026)
- [HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models](https://mlsanity.com/paper/2603.25411/) (2026)
- [LychSim: A Controllable and Interactive Simulation Framework for Vision Research](https://mlsanity.com/paper/2605.12449/) (2026)
- [Why Far Looks Up: Probing Spatial Representation in Vision-Language Models](https://mlsanity.com/paper/2605.30161/) (2026)
- [SATURN: Symbolic Spatial Reasoning for Multi-Perspective Grounding](https://mlsanity.com/paper/2606.22694/) (2026)
- [Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks](https://mlsanity.com/paper/2509.02175/) (2025)
- [FoR-SALE: Frame of Reference-guided Spatial Adjustment in LLM-based Diffusion Editing](https://mlsanity.com/paper/2509.23452/) (2025)
- [Obstruction reasoning for robotic grasping](https://mlsanity.com/paper/2511.23186/) (2025)
- [Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning](https://mlsanity.com/paper/2604.06725/) (2026)
- [Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models](https://mlsanity.com/paper/2605.15792/) (2026)
- [PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning](https://mlsanity.com/paper/2605.19382/) (2026)
- [SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes](https://mlsanity.com/paper/2605.31148/) (2026)
- [TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs](https://mlsanity.com/paper/2606.26029/) (2026)
- [Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information](https://mlsanity.com/paper/2608.27417/) (2026)
- [SCP: Spatial Causal Prediction in Video](https://mlsanity.com/paper/2603.03944/) (2026)
- [Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding](https://mlsanity.com/paper/2604.13313/) (2026)

## Abstract (arXiv)

Although large multimodal models (LMMs) have demonstrated remarkable capabilities in visual scene interpretation and reasoning, their capacity for complex and precise 3-dimensional spatial reasoning remains uncertain. Existing benchmarks focus predominantly on 2D spatial understanding and lack a framework to comprehensively evaluate 6D spatial reasoning across varying complexities. To address this limitation, we present Spatial457, a scalable and unbiased synthetic dataset designed with 4 key capability for spatial reasoning: multi-object recognition, 2D location, 3D location, and 3D orientation. We develop a cascading evaluation structure, constructing 7 question types across 5 difficulty levels that range from basic single object recognition to our new proposed complex 6D spatial reasoning tasks. We evaluated various large multimodal models (LMMs) on PulseCheck457, observing a general decline in performance as task complexity increases, particularly in 3D reasoning and 6D spatial tasks. To quantify these challenges, we introduce the Relative Performance Dropping Rate (RPDR), highlighting key weaknesses in 3D reasoning capabilities. Leveraging the unbiased attribute design of our dataset, we also uncover prediction biases across different attributes, with similar patterns observed in real-world image settings. The code and data are released in https://github.com/XingruiWang/Spatial457.