# Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Space

arXiv: https://arxiv.org/abs/2503.11094 · HTML: https://mlsanity.com/paper/2503.11094/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models](https://mlsanity.com/paper/2406.05756/) (2024)
- [KOSMOS-2.5: A Multimodal Literate Model](https://mlsanity.com/paper/2309.11419/) (2023)
- [Multi-modal Situated Reasoning in 3D Scenes](https://mlsanity.com/paper/2409.02389/) (2024)
- [An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models](https://mlsanity.com/paper/2411.06048/) (2024)
- [Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding](https://mlsanity.com/paper/2409.03757/) (2024)
- [EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment](https://mlsanity.com/paper/2410.09604/) (2024)
- [UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces](https://mlsanity.com/paper/2503.06157/) (2025)
- [SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models](https://mlsanity.com/paper/2411.13112/) (2024)

## Built on by (most cited first, 30 of 39)

- [WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent](https://mlsanity.com/paper/2508.05748/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views](https://mlsanity.com/paper/2510.18632/) (2025)
- [MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse](https://mlsanity.com/paper/2503.18470/) (2025)
- [Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods](https://mlsanity.com/paper/2511.15722/) (2025)
- [From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs](https://mlsanity.com/paper/2512.19683/) (2025)
- [SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagery](https://mlsanity.com/paper/2512.07733/) (2025)
- [CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environments](https://mlsanity.com/paper/2601.14339/) (2026)
- [Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis](https://mlsanity.com/paper/2510.08759/) (2025)
- [STReasoner: Empowering LLMs for Spatio-Temporal Reasoning in Time Series via Spatial-Aware Reinforcement Learning](https://mlsanity.com/paper/2601.03248/) (2026)
- [SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models](https://mlsanity.com/paper/2602.20901/) (2026)
- [SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments](https://mlsanity.com/paper/2604.22409/) (2026)
- [OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs](https://mlsanity.com/paper/2606.03890/) (2026)
- [USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents](https://mlsanity.com/paper/2505.17572/) (2025)
- [IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation](https://mlsanity.com/paper/2511.17384/) (2025)
- [ReasonX: MLLM-Guided Intrinsic Image Decomposition](https://mlsanity.com/paper/2512.04222/) (2025)
- [EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery](https://mlsanity.com/paper/2602.15918/) (2026)
- [Benchmarking Spatiotemporal Reasoning in LLMs and Reasoning Models: Capabilities and Challenges](https://mlsanity.com/paper/2505.11618/) (2025)
- [VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs](https://mlsanity.com/paper/2602.05382/) (2026)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspective](https://mlsanity.com/paper/2512.02340/) (2025)
- [ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models](https://mlsanity.com/paper/2603.13033/) (2026)
- [MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Models](https://mlsanity.com/paper/2507.23382/) (2025)
- [How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning](https://mlsanity.com/paper/2603.24866/) (2026)
- [ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models](https://mlsanity.com/paper/2605.20837/) (2026)
- [SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motion](https://mlsanity.com/paper/2606.27876/) (2026)
- [3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding](https://mlsanity.com/paper/2603.23447/) (2026)
- [Learning Multi-View Spatial Reasoning from Cross-View Relations](https://mlsanity.com/paper/2603.27967/) (2026)
- [GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understanding](https://mlsanity.com/paper/2607.13454/) (2026)
- [CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking](https://mlsanity.com/paper/2607.15004/) (2026)

## Abstract (arXiv)

Spatial reasoning is a fundamental capability of multimodal large language models (MLLMs), yet their performance in open aerial environments remains underexplored. In this work, we present Open3D-VQA, a novel benchmark for evaluating MLLMs' ability to reason about complex spatial relationships from an aerial perspective. The benchmark comprises 73k QA pairs spanning 7 general spatial reasoning tasks, including multiple-choice, true/false, and short-answer formats, and supports both visual and point cloud modalities. The questions are automatically generated from spatial relations extracted from both real-world and simulated aerial scenes. Evaluation on 13 popular MLLMs reveals that: 1) Models are generally better at answering questions about relative spatial relations than absolute distances, 2) 3D LLMs fail to demonstrate significant advantages over 2D LLMs, and 3) Fine-tuning solely on the simulated dataset can significantly improve the model's spatial reasoning performance in real-world scenarios. We release our benchmark, data generation pipeline, and evaluation toolkit to support further research: https://github.com/EmbodiedCity/Open3D-VQA.code.