UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Driving
On the map: LLM reasoning → Chain-of-thought reasoning for driving
also in 2 more
Abstract (arXiv)
Recent multimodal large language models (MLLMs) have shown strong potential for autonomous driving scene understanding, yet existing methods still face a fundamental trade-off between temporal reasoning and spatial precision. Models that rely on single-frame or low-resolution inputs often miss small, distant, or partially occluded hazards, while language-centric driving models frequently provide limited grounded evidence for their explanations. To address this gap, we propose UniDrive, a unified visual-language and grounding framework for interpretable risk understanding in autonomous driving. UniDrive combines a temporal reasoning branch that models scene dynamics from multi-frame visual input with a high-resolution perception branch that preserves fine-grained spatial details from the latest frame. The two branches are integrated through a gated cross-attention fusion module, enabling dynamic context to be aligned with precise spatial evidence. Based on the fused representation, UniDrive jointly generates natural-language risk descriptions and grounded bounding-box outputs for risk objects. Experiments on the DRAMA-Reasoning benchmark show that UniDrive outperforms representative image-based and video-based baselines in both captioning and risk-object grounding. In particular, UniDrive achieves the best overall performance on the validation split and demonstrates clear advantages in small-object localization, zero-shot generalization to NuScenes and BDD100K, and human-rated interpretability and trustworthiness. These results suggest that explicitly combining temporal semantics and high-resolution perception provides a stronger foundation for interpretable and safety-oriented autonomous driving systems. The code is available at https://github.com/pixeli99/unidrive-dev.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 21 of its 25 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
21 of 25 references.
Closest ancestors · 1
Same direction, newest first
Other references · 20
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
- 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
- 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
- 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
- 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
- 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
- 2023A Language Agent for Autonomous Drivingcited by 164
8 more
- 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
- 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
- 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
- 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
- 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
- 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 34
- 2026Are eHMIs always helpful? Investigating how eHMIs interfere with pedestrian behavior on multi-lane streets: An eye-tracking virtual reality experimentcited by 1
- 2026Wait or cross? Understanding the influence of behavioral tendency, trust, and risk perception on pedestrian gap-acceptance of automated truck platoonscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios
- 2026Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning
- 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
- 2025INSIGHT: Enhancing Autonomous Driving Safety through Vision-Language Models on Context-Aware Hazard Detection and Edge Case Evaluation
- 2026READ: Learning Risk-Informed Fields for End-to-End Autonomous Driving
- 2025Seeing before Observable: Potential Risk Reasoning in Autonomous Driving via Vision Language Models
- 2026ScenePilot-4K: A Large-Scale First-Person Dataset and Benchmark for Vision-Language Models in Autonomous Driving
- 2026CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning