# Seeing before Observable: Potential Risk Reasoning in Autonomous Driving via Vision Language Models

arXiv: https://arxiv.org/abs/2511.22928 · HTML: https://mlsanity.com/paper/2511.22928/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 28 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)
- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [SmolVLM: Redefining small and efficient multimodal models](https://mlsanity.com/paper/2504.05299/) (2025)
- [Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases](https://mlsanity.com/paper/2404.10595/) (2024)
- [T2SG: Traffic Topology Scene Graph for Topology Reasoning in Autonomous Driving](https://mlsanity.com/paper/2411.18894/) (2024)
- [Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark](https://mlsanity.com/paper/2212.09381/) (2022)
- [RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narratives](https://mlsanity.com/paper/2503.21459/) (2025)
- [DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models](https://mlsanity.com/paper/2505.07084/) (2025)
- [TruckV2X: A Truck-Centered Perception Dataset](https://mlsanity.com/paper/2507.09505/) (2025)

## Built on by (most cited first, 2 of 2)

- [VAGNet: Vision-based Accident Anticipation with Global Features](https://mlsanity.com/paper/2604.09305/) (2026)
- [Beyond Retrieval Relevance: Scene-Grounded Risk Entailment for Vision-Language Driving](https://mlsanity.com/paper/2609.34145/) (2026)

## Abstract (arXiv)

Ensuring safety remains a key challenge for autonomous vehicles (AVs), especially in rare and complex scenarios. One critical but understudied aspect is the potential risk situations, where the risk is not yet observable but can be inferred from subtle precursors, such as anomalous behaviors or commonsense violations. Recognizing these precursors requires strong semantic understanding and reasoning capabilities, which are often absent in current AV systems due to the scarcity of such cases in existing driving or risk-centric datasets. Moreover, current autonomous driving accident datasets often lack annotations of the causal reasoning chains behind incidents, which are essential for identifying potential risks before they become observable. To address these gaps, we introduce PotentialRiskQA, a novel vision-language dataset designed for reasoning about potential risks prior to observation. Each sample is annotated with structured scene descriptions, semantic precursors, and inferred risk outcomes. Based on this dataset, we further propose PR-Reasoner, a vision-language-model-based framework tailored for onboard potential risk reasoning. Experimental results show that fine-tuning on PotentialRiskQA enables PR-Reasoner to significantly enhance its performance on the potential risk reasoning task compared to baseline VLMs. Together, our dataset and model provide a foundation for developing autonomous systems with improved foresight and proactive safety capabilities, moving toward more intelligent and resilient AVs.