# UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web

arXiv: https://arxiv.org/abs/2310.18340 · HTML: https://mlsanity.com/paper/2310.18340/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 22 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Zero-Shot Video Question Answering via Frozen Bidirectional Language Models](https://mlsanity.com/paper/2206.08155/) (2022)
- [Tile2Vec: Unsupervised representation learning for spatially distributed data](https://mlsanity.com/paper/1805.02855/) (2018)
- [Using satellite imagery to understand and promote sustainable development](https://mlsanity.com/paper/2010.06988/) (2020)
- [Urban2Vec: Incorporating Street View Imagery and POIs for Multi-Modal Urban Neighborhood Embedding](https://mlsanity.com/paper/2001.11101/) (2020)
- [ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps](https://mlsanity.com/paper/2203.09127/) (2022)
- [Knowledge-infused Contrastive Learning for Urban Imagery-based Socioeconomic Prediction](https://mlsanity.com/paper/2302.13094/) (2023)
- [Take a Look Around: Using Street View and Satellite Images to Estimate House Prices](https://mlsanity.com/paper/1807.07155/) (2018)
- [Lightweight and Robust Representation of Economic Scales from Satellite Imagery](https://mlsanity.com/paper/1912.08197/) (2019)
- [Mutual Distillation Learning Network for Trajectory-User Linking](https://mlsanity.com/paper/2205.03773/) (2022)
- [MGeo: Multi-Modal Geographic Pre-Training Method](https://mlsanity.com/paper/2301.04283/) (2023)
- [Learning Economic Indicators by Aggregating Multi-Level Geospatial Information](https://mlsanity.com/paper/2205.01472/) (2022)

## Built on by (most cited first, 30 of 78)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [LLM Agents for Education: Advances and Applications](https://mlsanity.com/paper/2503.11733/) (2025)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning](https://mlsanity.com/paper/2502.12520/) (2025)
- [CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space](https://mlsanity.com/paper/2502.12532/) (2025)
- [FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs](https://mlsanity.com/paper/2407.02157/) (2024)
- [Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models](https://mlsanity.com/paper/2408.09429/) (2024)
- [MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2502.11051/) (2025)
- [UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios](https://mlsanity.com/paper/2408.17267/) (2024)
- [Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Models](https://mlsanity.com/paper/2506.14674/) (2025)
- [CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks](https://mlsanity.com/paper/2406.13945/) (2024)
- [Foundation Models for Spatio-Temporal Data Science: A Tutorial and Survey](https://mlsanity.com/paper/2503.13502/) (2025)
- [MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection](https://mlsanity.com/paper/2503.18132/) (2025)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook](https://mlsanity.com/paper/2503.11835/) (2025)
- [DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning](https://mlsanity.com/paper/2509.23883/) (2025)
- [From Text to Transformation: A Comprehensive Review of Large Language Models' Versatility](https://mlsanity.com/paper/2402.16142/) (2024)
- [UrbanVLP: Multi-Granularity Vision-Language Pretraining for Urban Socioeconomic Indicator Prediction](https://mlsanity.com/paper/2403.16831/) (2024)
- [VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models](https://mlsanity.com/paper/2504.13122/) (2025)
- [BIGCity: A Universal Spatiotemporal Model for Unified Trajectory and Traffic State Data Analysis](https://mlsanity.com/paper/2412.00953/) (2024)
- [On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applications](https://mlsanity.com/paper/2312.17016/) (2023)
- [FlexiReg: Flexible Urban Region Representation Learning](https://mlsanity.com/paper/2503.09128/) (2025)
- [GraphTeam: Facilitating Large Language Model-based Graph Analysis via Multi-Agent Collaboration](https://mlsanity.com/paper/2410.18032/) (2024)
- [A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science](https://mlsanity.com/paper/2504.09848/) (2025)
- [GeoReasoner: Reasoning On Geospatially Grounded Context For Natural Language Understanding](https://mlsanity.com/paper/2408.11366/) (2024)
- [POI-Enhancer: An LLM-based Semantic Enhancement Framework for POI Representation Learning](https://mlsanity.com/paper/2502.10038/) (2025)
- [Temporal Regularization Makes Your Video Generator Stronger](https://mlsanity.com/paper/2503.15417/) (2025)
- [Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Review](https://mlsanity.com/paper/2506.01364/) (2025)

## Abstract (arXiv)

Urban region profiling from web-sourced data is of utmost importance for urban planning and sustainable development. We are witnessing a rising trend of LLMs for various fields, especially dealing with multi-modal data research such as vision-language learning, where the text modality serves as a supplement information for the image. Since textual modality has never been introduced into modality combinations in urban region profiling, we aim to answer two fundamental questions in this paper: i) Can textual modality enhance urban region profiling? ii) and if so, in what ways and with regard to which aspects? To answer the questions, we leverage the power of Large Language Models (LLMs) and introduce the first-ever LLM-enhanced framework that integrates the knowledge of textual modality into urban imagery profiling, named LLM-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining (UrbanCLIP). Specifically, it first generates a detailed textual description for each satellite image by an open-source Image-to-Text LLM. Then, the model is trained on the image-text pairs, seamlessly unifying natural language supervision for urban visual representation learning, jointly with contrastive loss and language modeling loss. Results on predicting three urban indicators in four major Chinese metropolises demonstrate its superior performance, with an average improvement of 6.1% on R^2 compared to the state-of-the-art methods. Our code and the image-language dataset will be released upon paper notification.