# UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding

arXiv: https://arxiv.org/abs/2506.23219 · HTML: https://mlsanity.com/paper/2506.23219/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 29 Jun 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://mlsanity.com/paper/2310.05492/) (2023)
- [RSGPT: A Remote Sensing Vision Language Model and Benchmark](https://mlsanity.com/paper/2307.15266/) (2023)
- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model](https://mlsanity.com/paper/2402.02544/) (2024)
- [VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks](https://mlsanity.com/paper/2406.08394/) (2024)
- [Dolphins: Multimodal Language Model for Driving](https://mlsanity.com/paper/2312.00438/) (2023)
- [SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model](https://mlsanity.com/paper/2401.09712/) (2024)
- [SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding](https://mlsanity.com/paper/2406.10100/) (2024)
- [Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environment](https://mlsanity.com/paper/2312.11813/) (2023)
- [Mobility-LLM: Learning Visiting Intentions and Travel Preferences from Human Mobility Data with Large Language Models](https://mlsanity.com/paper/2411.00823/) (2024)
- [UrbanVLP: Multi-Granularity Vision-Language Pretraining for Urban Socioeconomic Indicator Prediction](https://mlsanity.com/paper/2403.16831/) (2024)
- [A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science](https://mlsanity.com/paper/2504.09848/) (2025)
- [LAMP: A Language Model on the Map](https://mlsanity.com/paper/2403.09059/) (2024)
- [CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing](https://mlsanity.com/paper/2506.00530/) (2025)

## Built on by (most cited first, 15 of 15)

- [CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environments](https://mlsanity.com/paper/2601.14339/) (2026)
- [UrbanFeel: A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective](https://mlsanity.com/paper/2509.22228/) (2025)
- [STReasoner: Empowering LLMs for Spatio-Temporal Reasoning in Time Series via Spatial-Aware Reinforcement Learning](https://mlsanity.com/paper/2601.03248/) (2026)
- [Semantic Edge-Cloud Communication for Real-Time Urban Traffic Surveillance with ViT and LLMs over Mobile Networks](https://mlsanity.com/paper/2509.21259/) (2025)
- [Urban-R1: Reinforced MLLMs Mitigate Geospatial Biases for Urban General Intelligence](https://mlsanity.com/paper/2510.16555/) (2025)
- [Improving Region Representation Learning from Urban Imagery with Noisy Long-Caption Supervision](https://mlsanity.com/paper/2511.07062/) (2025)
- [IoT-Brain: Grounding LLMs for Semantic-Spatial Sensor Scheduling](https://mlsanity.com/paper/2604.08033/) (2026)
- [ARMove: Learning to Predict Human Mobility through Agentic Reasoning](https://mlsanity.com/paper/2604.17419/) (2026)
- [Culturally uneven urban perception in large language models](https://mlsanity.com/paper/2604.20048/) (2026)
- [From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models](https://mlsanity.com/paper/2606.04381/) (2026)
- [Do VLMs See What Sensors Feel? A Scalable Expert-Guided Design for Wheelchair Accessibility Assessment from Street View](https://mlsanity.com/paper/2606.07642/) (2026)
- [UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City](https://mlsanity.com/paper/2608.27456/) (2026)
- [Algorithmic Shortlisting in Participatory Budgeting](https://mlsanity.com/paper/2508.06577/) (2025)
- [More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe](https://mlsanity.com/paper/2607.15942/) (2026)
- [UniGeo: A Multi-modal Large Language Model for Text-Guided Cross-View Geo-Localization](https://mlsanity.com/paper/2608.26722/) (2026)

## Abstract (arXiv)

Urban research involves a wide range of scenarios and tasks that require the understanding of multi-modal data. Current methods often focus on specific data types and lack a unified framework in urban field for processing them comprehensively. The recent success of multi-modal large language models (MLLMs) presents a promising opportunity to overcome this limitation. In this paper, we introduce textitUrbanLLaVA, a multi-modal large language model designed to process these four types of data simultaneously and achieve strong performance across diverse urban tasks compared with general MLLMs. In textitUrbanLLaVA, we first curate a diverse urban instruction dataset encompassing both single-modal and cross-modal urban data, spanning from location view to global view of urban environment. Additionally, we propose a multi-stage training framework that decouples spatial reasoning enhancement from domain knowledge learning, thereby improving the compatibility and downstream performance of textitUrbanLLaVA across diverse urban tasks. Finally, we also extend existing benchmark for urban research to assess the performance of MLLMs across a wide range of urban tasks. Experimental results from three cities demonstrate that textitUrbanLLaVA outperforms open-source and proprietary MLLMs in both single-modal tasks and complex cross-modal tasks and shows robust generalization abilities across cities. Source codes and data are openly accessible to the research community via https://github.com/tsinghua-fib-lab/UrbanLLaVA.