# SpatialLLM: From Multi-modality Data to Urban Spatial Intelligence

arXiv: https://arxiv.org/abs/2505.12703 · HTML: https://mlsanity.com/paper/2505.12703/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 30 of 32)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023)
- [Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning](https://mlsanity.com/paper/2403.11401/) (2024)
- [Stratified Transformer for 3D Point Cloud Segmentation](https://mlsanity.com/paper/2203.14508/) (2022)
- [Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes](https://mlsanity.com/paper/2308.08769/) (2023)
- [On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence](https://mlsanity.com/paper/2304.06798/) (2023)
- [Autonomous GIS: the next-generation AI-powered GIS](https://mlsanity.com/paper/2305.06453/) (2023)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPT](https://mlsanity.com/paper/2307.07930/) (2023)
- [MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations](https://mlsanity.com/paper/2406.09401/) (2024)
- [CityGPT: Empowering Urban Spatial Cognition of Large Language Models](https://mlsanity.com/paper/2406.13948/) (2024)
- [UrbanBIS: a Large-scale Benchmark for Fine-grained Urban Building Instance Segmentation](https://mlsanity.com/paper/2305.02627/) (2023)
- [CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Data](https://mlsanity.com/paper/2310.18773/) (2023)
- [Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluation](https://mlsanity.com/paper/2310.15676/) (2023)
- [SceneGPT: A Language Model for 3D Scene Understanding](https://mlsanity.com/paper/2408.06926/) (2024)
- [Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models](https://mlsanity.com/paper/2409.15451/) (2024)

## Built on by (most cited first, 8 of 8)

- [GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models](https://mlsanity.com/paper/2510.07791/) (2025)
- [RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios](https://mlsanity.com/paper/2511.18011/) (2025)
- [Spatial-Morphological Modeling for Multi-Attribute Imputation of Urban Blocks](https://mlsanity.com/paper/2602.10923/) (2026)
- [Evaluating Few-Shot Temporal Reasoning of LLMs for Human Activity Prediction in Smart Environments](https://mlsanity.com/paper/2602.11176/) (2026)
- [Ground-Truth Depth in Vision Language Models: Spatial Context Understanding in Conversational AI for XR-Robotic Support in Emergency First Response](https://mlsanity.com/paper/2602.15237/) (2026)
- [UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics](https://mlsanity.com/paper/2606.15890/) (2026)
- [LifePlanner: Evaluating LLM Agents for Geo-spatial Planning with Social Media Data](https://mlsanity.com/paper/2608.25039/) (2026)
- [CitySTAR: Structured and Topology-Aware Reasoning for Open-Vocabulary Urban 3D Grounding](https://mlsanity.com/paper/2609.19911/) (2026)

## Abstract (arXiv)

We propose SpatialLLM, a novel approach advancing spatial intelligence tasks in complex urban scenes. Unlike previous methods requiring geographic analysis tools or domain expertise, SpatialLLM is a unified language model directly addressing various spatial intelligence tasks without any training, fine-tuning, or expert intervention. The core of SpatialLLM lies in constructing detailed and structured scene descriptions from raw spatial data to prompt pre-trained LLMs for scene-based analysis. Extensive experiments show that, with our designs, pretrained LLMs can accurately perceive spatial distribution information and enable zero-shot execution of advanced spatial intelligence tasks, including urban planning, ecological analysis, traffic management, etc. We argue that multi-field knowledge, context length, and reasoning ability are key factors influencing LLM performances in urban analysis. We hope that SpatialLLM will provide a novel viable perspective for urban intelligent analysis and management. The code and dataset are available at https://github.com/WHU-USI3DV/SpatialLLM.