AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
This research investigates the potential of Vision-Language Models (VLMs) to infer building typologies: Construction, Current Use, and Storeys from Google Street View (GSV) images. Predictions generated by VLMs are compared with inference by human experts (civil engineers and architects) as a source of manually labelled ground-truth data. We evaluate several state-of-the-art VLMs, including GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash. By applying different scaling strategies and prompting techniques, we found that Chain-of-Thought prompts provide an overall more stable model performance. We also investigate the reasoning behind VLMs' building-typology predictions by examining the probabilities of keywords appearing in AI explanations. This enabled us to analyse patterns in these reasonings and identify key themes driving both agreements and disagreements between VLM and expert labels. We find that AI tends to focus on visual indicators, whereas human experts place greater emphasis on broader contextual cues and domain knowledge, in addition to visual cues. Overall, VLM can approximate experts' capability in building-typology classification at scale, with an average accuracy of approximately 70%. The study demonstrates the VLM's potential for AI automation in tasks that require pattern recognition and object identification in an urban context. AI have the potential to serve as complementary and collaborative tools for urban analysis, leveraging their strengths in understanding visual patterns. This study contributes to the exploration of the efficiency and scalability of AI visual prediction and provides insights into the reasoning processes that could support automation processes in urban analysis and prediction.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 10 of its 43 references that are in the map. First posted on arXiv 16 Jul 2026.
Builds on
10 of 43 references.
References · 10
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
- 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
- 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
- 2024Coverage and Bias of Street View Imagery in Mapping the Urban Environmentcited by 11
- 2018Estimating city-level travel patterns using street imagery: a case study of using Google Street View in Britaincited by 10
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings
- 2026Leveraging Multimodal LLMs for Built Environment and Housing Attribute Assessment from Street-View Imagery
- 2025Can a Large Language Model Assess Urban Design Quality? Evaluating Walkability Metrics Across Expertise Levels
- 2026USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments
- 2026Built Environment Reasoning from Remote Sensing Imagery Using Large Vision--Language Models
- 2025Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models
- 2026Can Large Multimodal Models Inspect Buildings? A Hierarchical Benchmark for Structural Pathology Reasoning
- 2026How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning