# Urban Safety Perception Assessments via Integrating Multimodal Large Language Models with Street View Images

arXiv: https://arxiv.org/abs/2407.19719 · HTML: https://mlsanity.com/paper/2407.19719/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 61 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [Deep Learning the City : Quantifying Urban Perception At A Global Scale](https://mlsanity.com/paper/1608.01769/) (2016)
- [Learning Bottleneck Concepts in Image Classification](https://mlsanity.com/paper/2304.10131/) (2023)
- [TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction](https://mlsanity.com/paper/2310.15556/) (2023)
- [ChatGPT: Applications, Opportunities, and Threats](https://mlsanity.com/paper/2304.09103/) (2023)
- [Adaptive Contrast for Image Regression in Computer-Aided Disease Assessment](https://mlsanity.com/paper/2112.11700/) (2021)
- [Semi-Supervised Deep Regression with Uncertainty Consistency and Variational Model Ensembling via Bayesian Neural Networks](https://mlsanity.com/paper/2302.07579/) (2023)
- [A citizen science toolkit to collect human perceptions of urban environments using open street view images](https://mlsanity.com/paper/2403.00174/) (2024)
- [Improving Facade Parsing with Vision Transformers and Line Integration](https://mlsanity.com/paper/2309.15523/) (2023)
- [Match Them Up: Visually Explainable Few-shot Image Classification](https://mlsanity.com/paper/2011.12527/) (2020)

## Built on by (most cited first, 8 of 8)

- [UrbanFeel: A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective](https://mlsanity.com/paper/2509.22228/) (2025)
- [Multi-Modal Feature Fusion for Spatial Morphology Analysis of Traditional Villages via Hierarchical Graph Neural Networks](https://mlsanity.com/paper/2510.27208/) (2025)
- [Culturally uneven urban perception in large language models](https://mlsanity.com/paper/2604.20048/) (2026)
- [From Drone Imagery to Livability Mapping: AI-powered Environment Perception in Rural China](https://mlsanity.com/paper/2508.21738/) (2025)
- [URBAN-SPIN: A street-level bikeability index to inform design implementations in historical city centres](https://mlsanity.com/paper/2602.10124/) (2026)
- [UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics](https://mlsanity.com/paper/2606.15890/) (2026)
- [Caption-Mediated Perceived-Safety Estimation for Pedestrian Routing](https://mlsanity.com/paper/2609.38479/) (2026)
- [FearCaut-Qwen: Affective Steering in a Vision-Language Model Shifts the Decision Criterion for Hazard Assessment](https://mlsanity.com/paper/2610.11986/) (2026)

## Abstract (arXiv)

Measuring urban safety perception is an important and complex task that traditionally relies heavily on human resources. This process often involves extensive field surveys, manual data collection, and subjective assessments, which can be time-consuming, costly, and sometimes inconsistent. Street View Images (SVIs), along with deep learning methods, provide a way to realize large-scale urban safety detection. However, achieving this goal often requires extensive human annotation to train safety ranking models, and the architectural differences between cities hinder the transferability of these models. Thus, a fully automated method for conducting safety evaluations is essential. Recent advances in multimodal large language models (MLLMs) have demonstrated powerful reasoning and analytical capabilities. Cutting-edge models, e.g., GPT-4 have shown surprising performance in many tasks. We employed these models for urban safety ranking on a human-annotated anchor set and validated that the results from MLLMs align closely with human perceptions. Additionally, we proposed a method based on the pre-trained Contrastive Language-Image Pre-training (CLIP) feature and K-Nearest Neighbors (K-NN) retrieval to quickly assess the safety index of the entire city. Experimental results show that our method outperforms existing training needed deep learning approaches, achieving efficient and accurate urban safety evaluations. The proposed automation for urban safety perception assessment is a valuable tool for city planners, policymakers, and researchers aiming to improve urban environments.