# CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks

arXiv: https://arxiv.org/abs/2406.13945 · HTML: https://mlsanity.com/paper/2406.13945/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 32 of its 62 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 32)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization](https://mlsanity.com/paper/2309.16020/) (2023)
- [Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model](https://mlsanity.com/paper/2208.03987/) (2022)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario](https://mlsanity.com/paper/1905.05217/) (2019)
- [Learning to Navigate in Cities Without a Map](https://mlsanity.com/paper/1804.00168/) (2018)
- [Where Would I Go Next? Large Language Models as Human Mobility Predictors](https://mlsanity.com/paper/2308.15197/) (2023)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence](https://mlsanity.com/paper/2304.06798/) (2023)
- [A Survey on Traffic Signal Control Methods](https://mlsanity.com/paper/1904.08117/) (2019)
- [UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web](https://mlsanity.com/paper/2310.18340/) (2023)
- [VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View](https://mlsanity.com/paper/2307.06082/) (2023)
- [Can Language Models Serve as Text-Based World Simulators?](https://mlsanity.com/paper/2406.06485/) (2024)
- [V-IRL: Grounding Virtual Intelligence in Real Life](https://mlsanity.com/paper/2402.03310/) (2024)
- [Are Large Language Models Geospatially Knowledgeable?](https://mlsanity.com/paper/2310.13002/) (2023)

## Built on by (most cited first, 27 of 27)

- [AgentMove: A Large Language Model based Agentic Framework for Zero-shot Next Location Prediction](https://mlsanity.com/paper/2408.13986/) (2024)
- [ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps](https://mlsanity.com/paper/2505.18675/) (2025)
- [Grounded Vision-Language Navigation for UAVs with Open-Vocabulary Goal Understanding](https://mlsanity.com/paper/2506.10756/) (2025)
- [Generative AI in Transportation Planning: A Survey](https://mlsanity.com/paper/2503.07158/) (2025)
- [CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing](https://mlsanity.com/paper/2506.00530/) (2025)
- [UrbanFeel: A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective](https://mlsanity.com/paper/2509.22228/) (2025)
- [LocalGPT: Benchmarking and Advancing Large Language Models for Local Life Services in Meituan](https://mlsanity.com/paper/2506.02720/) (2025)
- [Urban Computing in the Era of Large Language Models](https://mlsanity.com/paper/2504.02009/) (2025)
- [Satellites Reveal Mobility: A Commuting Origin-destination Flow Generator for Global Cities](https://mlsanity.com/paper/2505.15870/) (2025)
- [Towards General Urban Monitoring with Vision-Language Models: A Review, Evaluation, and a Research Agenda](https://mlsanity.com/paper/2510.12400/) (2025)
- [Learning to Factorize and Adapt: A Versatile Approach Toward Universal Spatio-Temporal Foundation Models](https://mlsanity.com/paper/2601.12083/) (2026)
- [Whose Name Comes Up? II: Benchmarking and Intervention-Based Auditing of LLM-Based Scholar Recommendation](https://mlsanity.com/paper/2602.08873/) (2026)
- [DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum](https://mlsanity.com/paper/2609.19801/) (2026)
- [Chat2SPaT: A Large Language Model Based Tool for Automating Traffic Signal Control Plan Management](https://mlsanity.com/paper/2507.05283/) (2025)
- [360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents](https://mlsanity.com/paper/2608.08814/) (2026)
- [Open-Set Living Need Prediction with Large Language Models](https://mlsanity.com/paper/2506.02713/) (2025)
- [Algorithmic Shortlisting in Participatory Budgeting](https://mlsanity.com/paper/2508.06577/) (2025)
- [TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route](https://mlsanity.com/paper/2509.18173/) (2025)
- [StreetTree: A Large-Scale Global Benchmark for Fine-Grained Tree Species Classification](https://mlsanity.com/paper/2602.19123/) (2026)
- [TransitLM: A Large-Scale Dataset and Benchmark for Map-Free Transit Route Generation](https://mlsanity.com/paper/2605.22355/) (2026)
- [UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics](https://mlsanity.com/paper/2606.15890/) (2026)
- [GenWorld: Empirically Grounded Urban Simulation Infrastructure for Scalable LLM-Agent Studies](https://mlsanity.com/paper/2606.27650/) (2026)
- [UrbanDS: A Graph-Guided LLM Multi-Agent System for Data-Intensive Urban Tasks](https://mlsanity.com/paper/2607.26724/) (2026)
- [Mapping the City Through the Lens of Language Models](https://mlsanity.com/paper/2608.02971/) (2026)
- [UrbanAgent: A Tool-Augmented Agent for Cross-System Urban Tasks](https://mlsanity.com/paper/2608.03018/) (2026)
- [Do LLMs Know Your Neighborhood? Auditing LLM Priors for Neighborhood-Level Mobility Prediction and Structural Alignment](https://mlsanity.com/paper/2609.00345/) (2026)
- [CityCode: Building Agent-Operable 3D Cities from Sparse Aerial Observations](https://mlsanity.com/paper/2610.18342/) (2026)

## Abstract (arXiv)

As large language models (LLMs) continue to advance and gain widespread use, establishing systematic and reliable evaluation methodologies for LLMs and vision-language models (VLMs) has become essential to ensure their real-world effectiveness and reliability. There have been some early explorations about the usability of LLMs for limited urban tasks, but a systematic and scalable evaluation benchmark is still lacking. The challenge in constructing a systematic evaluation benchmark for urban research lies in the diversity of urban data, the complexity of application scenarios and the highly dynamic nature of the urban environment. In this paper, we design textitCityBench, an interactive simulator based evaluation platform, as the first systematic benchmark for evaluating the capabilities of LLMs for diverse tasks in urban research. First, we build textitCityData to integrate the diverse urban data and textitCitySimu to simulate fine-grained urban dynamics. Based on textitCityData and textitCitySimu, we design 8 representative urban tasks in 2 categories of perception-understanding and decision-making as the textitCityBench. With extensive results from 30 well-known LLMs and VLMs in 13 cities around the world, we find that advanced LLMs and VLMs can achieve competitive performance in diverse urban tasks requiring commonsense and semantic understanding abilities, e.g., understanding the human dynamics and semantic inference of urban images. Meanwhile, they fail to solve the challenging urban tasks requiring professional knowledge and high-level numerical abilities, e.g., geospatial prediction and traffic control task.