mlsanityData to 7 Oct 2026

Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

arXiv 2606.00871 ↗First posted 30 May 2026cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models (VLMs) are increasingly used to generate structured descriptions of street-level imagery for tasks such as streetscape auditing, mapping, and public consultation. These uses combine observable attributes with appraisal categories, and the human targets are often distributions of judgments with disagreement and explicit non-response. This paper argues that benchmarking VLMs for urban perception should treat disagreement and abstention as measurement outcomes, report inter-annotator reliability alongside model alignment, and treat the label space and scoring policy as negotiable artifacts when outputs are intended to inform urban governance. We ground the argument in a benchmark of 100 Montreal street scenes annotated along 30 dimensions by 12 participants from seven community organizations, and in a deterministic zero-shot evaluation of seven VLMs. Across dimensions, model agreement with human consensus co-varies with dimension-level human reliability, and for the appraisal dimension Overall Impression models and annotators exhibit distributional mismatch including different rates of Not applicable. We close with actions for benchmark creators, model developers, and institutions to make uncertainty and benchmark assumptions visible in evaluation reports.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 31 references that are in the map. First posted on arXiv 30 May 2026.

Builds on

14 of 31 references.

References · 14

  1. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  2. 2018Datasheets for Datasetscited by 1,371
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  4. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  5. 2018Learning to Navigate in Cities Without a Mapcited by 103
  6. 2017Runaway Feedback Loops in Predictive Policingcited by 97
  7. 2024Large Language Models are Geographically Biasedcited by 91
  8. 2023UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Webcited by 77
  9. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  10. 2016Deep Learning the City : Quantifying Urban Perception At A Global Scalecited by 65
  11. 2023Are Large Language Models Geospatially Knowledgeable?cited by 30
  12. 2024UrbanVLP: Multi-Granularity Vision-Language Pretraining for Urban Socioeconomic Indicator Predictioncited by 19
2 more
  1. 2025LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spacescited by 8
  2. 2025Co-Producing AI: Toward an Augmented, Participatory Lifecyclecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026You Cannot Photograph the Same Street Twice: Reliability Limits in Vision-Language Measurement of Urban Changecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Vision-Language Models See Urban Scenes as People Do? An Urban Perception Benchmark
  2. 2026UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment
  3. 2025Multimodal Large Language Models Predict Urban Safety Perception but Encode Non-Neutral Demographic Priors
  4. 2025Interpretable Multimodal Framework for Human-Centered Street Assessment: Integrating Visual-Language Models for Perceptual Urban Diagnostics
  5. 2025UrbanFeel: A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective
  6. 2024CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks
  7. 2026From Street View Imagery to Street Quality Indicators: Vision Language Inference for the Suburban 15-minute City
  8. 2025StreetLens: Enabling Human-Centered AI Agents for Neighborhood Assessment from Street View Imagery