# Do 3D Large Language Models Really Understand 3D Spatial Relationships?

arXiv: https://arxiv.org/abs/2603.23523 · HTML: https://mlsanity.com/paper/2603.23523/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 30 of 32)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [PointLLM: Empowering Large Language Models to Understand Point Clouds](https://mlsanity.com/paper/2308.16911/) (2023)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following](https://mlsanity.com/paper/2309.00615/) (2023)
- [LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness](https://mlsanity.com/paper/2409.18125/) (2024)
- [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023)
- [Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning](https://mlsanity.com/paper/2403.11401/) (2024)
- [LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token](https://mlsanity.com/paper/2501.03895/) (2025)
- [Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding](https://mlsanity.com/paper/2412.00493/) (2024)
- [LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent](https://mlsanity.com/paper/2309.12311/) (2023)
- [Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes](https://mlsanity.com/paper/2308.08769/) (2023)
- [GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models](https://mlsanity.com/paper/2501.01428/) (2025)
- [LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding](https://mlsanity.com/paper/2312.14074/) (2023)
- [Vision Language Models are Biased](https://mlsanity.com/paper/2505.23941/) (2025)
- [Multi-modal Situated Reasoning in 3D Scenes](https://mlsanity.com/paper/2409.02389/) (2024)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [3D Question Answering](https://mlsanity.com/paper/2112.08359/) (2021)
- [M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts](https://mlsanity.com/paper/2312.10763/) (2023)
- [3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding](https://mlsanity.com/paper/2401.03201/) (2024)
- [SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models](https://mlsanity.com/paper/2410.03878/) (2024)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis](https://mlsanity.com/paper/2503.22420/) (2025)

## Built on by (most cited first, 9 of 9)

- [Analogical Trajectory Transfer](https://mlsanity.com/paper/2605.14393/) (2026)
- [Aggregating LLM-Based Weak Verifiers for Spatial Layout Generation](https://mlsanity.com/paper/2606.05268/) (2026)
- [Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering](https://mlsanity.com/paper/2609.38177/) (2026)
- [OneCanvas: 3D Scene Understanding via Panoramic Reprojection](https://mlsanity.com/paper/2606.19253/) (2026)
- [IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning](https://mlsanity.com/paper/2607.03614/) (2026)
- [Do LiDAR Language Models Really Understand Spatio-temporal Relationships?](https://mlsanity.com/paper/2609.24452/) (2026)
- [EgoAlign: Situated 3D Understanding without Situation Descriptions](https://mlsanity.com/paper/2610.16176/) (2026)
- [What Makes a Good Visual Representation for Spatial Understanding in MLLMs?](https://mlsanity.com/paper/2610.17844/) (2026)
- [Can Learning to Answer Spatial Questions Improve Novel-View Generation?](https://mlsanity.com/paper/2610.20610/) (2026)

## Abstract (arXiv)

Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, we find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not be able to detect if the model exploits textual shortcuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning. Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that guides model to rely more on 3D visual clues, substantially enhancing 3D-LLMs performance in spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding. Project page: https://real-3dqa.github.io/.