# Do Vision-Language Models See Urban Scenes as People Do? An Urban Perception Benchmark

arXiv: https://arxiv.org/abs/2509.14574 · HTML: https://mlsanity.com/paper/2509.14574/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 13 of 13)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Deep Learning the City : Quantifying Urban Perception At A Global Scale](https://mlsanity.com/paper/1608.01769/) (2016)
- [LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces](https://mlsanity.com/paper/2503.01894/) (2025)
- [Negotiative Alignment: Embracing Disagreement to Achieve Fairer Outcomes -- Insights from Urban Studies](https://mlsanity.com/paper/2503.12613/) (2025)
- [The Right to AI](https://mlsanity.com/paper/2501.17899/) (2025)
- [Street Review: A Participatory AI-Based Framework for Assessing Streetscape Inclusivity](https://mlsanity.com/paper/2508.11708/) (2025)
- [WeDesign: Generative AI-Facilitated Community Consultations for Urban Public Space Design](https://mlsanity.com/paper/2508.19256/) (2025)
- [Prompt Commons: Collective Prompting as Governance for Urban AI](https://mlsanity.com/paper/2509.12415/) (2025)
- [Urban AI Governance Must Embed Legal Reasonableness for Democratic and Sustainable Cities](https://mlsanity.com/paper/2508.12174/) (2025)
- [Collective Recourse for Generative Urban Visualizations](https://mlsanity.com/paper/2509.11487/) (2025)

## Built on by (most cited first, 4 of 4)

- [UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment](https://mlsanity.com/paper/2602.19442/) (2026)
- [Culturally uneven urban perception in large language models](https://mlsanity.com/paper/2604.20048/) (2026)
- [AI Pluralism and the Worlds It Misses](https://mlsanity.com/paper/2606.16167/) (2026)
- [From Street View Imagery to Street Quality Indicators: Vision Language Inference for the Suburban 15-minute City](https://mlsanity.com/paper/2608.20026/) (2026)

## Abstract (arXiv)

Understanding how people read city scenes can inform design and planning. We introduce a small benchmark for testing vision-language models (VLMs) on urban perception using 100 Montreal street images, evenly split between photographs and photorealistic synthetic scenes. Twelve participants from seven community groups supplied 230 annotation forms across 30 dimensions mixing physical attributes and subjective impressions. French responses were normalized to English. We evaluated seven VLMs in a zero-shot setup with a structured prompt and deterministic parser. We use accuracy for single-choice items and Jaccard overlap for multi-label items; human agreement uses Krippendorff's alpha and pairwise Jaccard. Results suggest stronger model alignment on visible, objective properties than subjective appraisals. The top system (claude-sonnet) reaches macro 0.31 and mean Jaccard 0.48 on multi-label items. Higher human agreement coincides with better model scores. Synthetic images slightly lower scores. We release the benchmark, prompts, and harness for reproducible, uncertainty-aware evaluation in participatory urban analysis.