# How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM

arXiv: https://arxiv.org/abs/2504.05786 · HTML: https://mlsanity.com/paper/2504.05786/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 19 of 19)

- [Uni3D: Exploring Unified 3D Representation at Scale](https://mlsanity.com/paper/2310.06773/) (2023)
- [Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following](https://mlsanity.com/paper/2309.00615/) (2023)
- [LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness](https://mlsanity.com/paper/2409.18125/) (2024)
- [Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning](https://mlsanity.com/paper/2403.11401/) (2024)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes](https://mlsanity.com/paper/2308.08769/) (2023)
- [Grounded 3D-LLM with Referent Tokens](https://mlsanity.com/paper/2405.10370/) (2024)
- [LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding](https://mlsanity.com/paper/2312.14074/) (2023)
- [3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer](https://mlsanity.com/paper/2501.01163/) (2025)
- [SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors](https://mlsanity.com/paper/2403.13438/) (2024)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment](https://mlsanity.com/paper/2410.09604/) (2024)
- [Uni3D-LLM: Unifying Point Cloud Perception, Generation and Editing with Large Language Models](https://mlsanity.com/paper/2402.03327/) (2024)
- [3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models](https://mlsanity.com/paper/2409.19330/) (2024)
- [Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation](https://mlsanity.com/paper/2406.07146/) (2024)
- [More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding](https://mlsanity.com/paper/2408.15966/) (2024)
- [Kestrel: 3D Multimodal LLM for Part-Aware Grounded Description](https://mlsanity.com/paper/2405.18937/) (2024)
- [LLMI3D: MLLM-based 3D Perception from a Single 2D Image](https://mlsanity.com/paper/2408.07422/) (2024)
- [OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicine](https://mlsanity.com/paper/2402.18028/) (2024)

## Built on by (most cited first, 30 of 42)

- [ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models](https://mlsanity.com/paper/2505.21500/) (2025)
- [Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views](https://mlsanity.com/paper/2510.18632/) (2025)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2510.27606/) (2025)
- [D3: Training-Free AI-Generated Video Detection Using Second-Order Features](https://mlsanity.com/paper/2508.00701/) (2025)
- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [Code2Video: A Code-centric Paradigm for Educational Video Generation](https://mlsanity.com/paper/2510.01174/) (2025)
- [Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization](https://mlsanity.com/paper/2505.22038/) (2025)
- [Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes](https://mlsanity.com/paper/2510.19400/) (2025)
- [AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoning](https://mlsanity.com/paper/2511.11025/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagery](https://mlsanity.com/paper/2512.07733/) (2025)
- [AirScape: An Aerial Generative World Model with Motion Controllability](https://mlsanity.com/paper/2507.08885/) (2025)
- [VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments](https://mlsanity.com/paper/2506.02387/) (2025)
- [AInsteinBench: Benchmarking Coding Agents on Scientific Repositories](https://mlsanity.com/paper/2512.21373/) (2025)
- [Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation](https://mlsanity.com/paper/2602.11635/) (2026)
- [S2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance](https://mlsanity.com/paper/2512.01223/) (2025)
- [PanoWorld: Towards Spatial Supersensing in 360^circ Panorama World](https://mlsanity.com/paper/2605.13169/) (2026)
- [Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts](https://mlsanity.com/paper/2505.21079/) (2025)
- [ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Models](https://mlsanity.com/paper/2605.10106/) (2026)
- [EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy](https://mlsanity.com/paper/2605.24456/) (2026)
- [Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions](https://mlsanity.com/paper/2507.14555/) (2025)
- [MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Images](https://mlsanity.com/paper/2511.19119/) (2025)
- [Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling](https://mlsanity.com/paper/2512.01821/) (2025)
- [SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning](https://mlsanity.com/paper/2512.16461/) (2025)
- [How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning](https://mlsanity.com/paper/2603.24866/) (2026)
- [OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping](https://mlsanity.com/paper/2607.00881/) (2026)
- [Context-Aware Sentiment Forecasting via LLM-based Multi-Perspective Role-Playing Agents](https://mlsanity.com/paper/2505.24331/) (2025)
- [From Pixels to Policies: Reinforcing Spatial Reasoning in Language Models for Content-Aware Layout Design](https://mlsanity.com/paper/2602.13912/) (2026)
- [Learning Multi-View Spatial Reasoning from Cross-View Relations](https://mlsanity.com/paper/2603.27967/) (2026)

## Abstract (arXiv)

3D spatial understanding is essential in real-world applications such as robotics, autonomous vehicles, virtual reality, and medical imaging. Recently, Large Language Models (LLMs), having demonstrated remarkable success across various domains, have been leveraged to enhance 3D understanding tasks, showing potential to surpass traditional computer vision methods. In this survey, we present a comprehensive review of methods integrating LLMs with 3D spatial understanding. We propose a taxonomy that categorizes existing methods into three branches: image-based methods deriving 3D understanding from 2D visual data, point cloud-based methods working directly with 3D representations, and hybrid modality-based methods combining multiple data streams. We systematically review representative methods along these categories, covering data representations, architectural modifications, and training strategies that bridge textual and 3D modalities. Finally, we discuss current limitations, including dataset scarcity and computational challenges, while highlighting promising research directions in spatial perception, multi-modal fusion, and real-world applications.