# Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing arXiv: https://arxiv.org/abs/2402.06794 ยท HTML: https://mlsanity.com/paper/2402.06794/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 25 of its 40 references that are in the map. - First posted on arXiv 9 Feb 2024. ## Placed in - Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/) ## Builds on (most cited first, 25 of 25) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023) - [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016) - [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023) - [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023) - [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020) - [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018) - [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023) - [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023) - [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023) - [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023) - [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023) - [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023) - [Towards Robust Robot 3D Perception in Urban Environments: The UT Campus Object Dataset](https://mlsanity.com/paper/2309.13549/) (2023) - [Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing](https://mlsanity.com/paper/1808.06887/) (2018) - [Breaking the Bank with ChatGPT: Few-Shot Text Classification for Finance](https://mlsanity.com/paper/2308.14634/) (2023) - [DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Grounding](https://mlsanity.com/paper/2307.06924/) (2023) - [Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information](https://mlsanity.com/paper/2305.01788/) (2023) ## Built on by (most cited first, 12 of 12) - [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025) - [AVD2: Accident Video Diffusion for Accident Video Description](https://mlsanity.com/paper/2502.14801/) (2025) - [VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation](https://mlsanity.com/paper/2403.12415/) (2024) - [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025) - [Space-Aware Instruction Tuning: Dataset and Benchmark for Guide Dog Robots Assisting the Visually Impaired](https://mlsanity.com/paper/2502.07183/) (2025) - [NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving](https://mlsanity.com/paper/2509.25944/) (2025) - [StreetSurfaceVis: a dataset of crowdsourced street-level imagery annotated by road surface type and quality](https://mlsanity.com/paper/2407.21454/) (2024) - [Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation](https://mlsanity.com/paper/2410.14975/) (2024) - [Gen-AI for User Safety: A Survey](https://mlsanity.com/paper/2411.06606/) (2024) - [WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation](https://mlsanity.com/paper/2603.10703/) (2026) - [Urban Risk-Aware Navigation via VQA-Based Event Maps for People with Low Vision](https://mlsanity.com/paper/2605.11782/) (2026) - [Synthetic data augmentation for robotic mobility aids to support blind and low vision people](https://mlsanity.com/paper/2409.11164/) (2024) ## Abstract (arXiv) Safely navigating street intersections is a complex challenge for blind and low-vision individuals, as it requires a nuanced understanding of the surrounding context - a task heavily reliant on visual cues. Traditional methods for assisting in this decision-making process often fall short, lacking the ability to provide a comprehensive scene analysis and safety level. This paper introduces an innovative approach that leverages large multimodal models (LMMs) to interpret complex street crossing scenes, offering a potential advancement over conventional traffic signal recognition techniques. By generating a safety score and scene description in natural language, our method supports safe decision-making for the blind and low-vision individuals. We collected crosswalk intersection data that contains multiview egocentric images captured by a quadruped robot and annotated the images with corresponding safety scores based on our predefined safety score categorization. Grounded on the visual knowledge, extracted from images, and text prompt, we evaluate a large multimodal model for safety score prediction and scene description. Our findings highlight the reasoning and safety score prediction capabilities of a LMM, activated by various prompts, as a pathway to developing a trustworthy system, crucial for applications requiring reliable decision-making support.