# SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Models

arXiv: https://arxiv.org/abs/2412.13238 · HTML: https://mlsanity.com/paper/2412.13238/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving.
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- End-to-end autonomous driving / [Adversarial scenarios and safety testing](https://mlsanity.com/driving/adversarial-scenarios-and-safety-testing/)

## Builds on (most cited first, 17 of 17)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024)
- [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023)
- [Drive Like a Human: Rethinking Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2307.07162/) (2023)
- [The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections](https://mlsanity.com/paper/1911.07602/) (2019)
- [On a Formal Model of Safe and Scalable Self-driving Cars](https://mlsanity.com/paper/1708.06374/) (2017)
- [Neural-Fly Enables Rapid Learning for Agile Flight in Strong Winds](https://mlsanity.com/paper/2205.06908/) (2022)
- [Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion](https://mlsanity.com/paper/2401.17583/) (2024)
- [Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2310.08034/) (2023)
- [Real-Time Anomaly Detection and Reactive Planning with Large Language Models](https://mlsanity.com/paper/2407.08735/) (2024)
- [A Comparative Study on Reasoning Patterns of OpenAI's o1 Model](https://mlsanity.com/paper/2410.13639/) (2024)
- [When Large Language Models Meet Vector Databases: A Survey](https://mlsanity.com/paper/2402.01763/) (2024)
- [Towards Knowledge-driven Autonomous Driving](https://mlsanity.com/paper/2312.04316/) (2023)
- [Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF](https://mlsanity.com/paper/2406.04481/) (2024)
- [KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2407.14239/) (2024)

## Built on by (most cited first, 10 of 10)

- [Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects](https://mlsanity.com/paper/2505.20223/) (2025)
- [OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic](https://mlsanity.com/paper/2512.01830/) (2025)
- [A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving](https://mlsanity.com/paper/2607.07103/) (2026)
- [Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving](https://mlsanity.com/paper/2501.06680/) (2025)
- [Case-based Reasoning Augmented Large Language Model Framework for Decision Making in Realistic Safety-Critical Driving Scenarios](https://mlsanity.com/paper/2506.20531/) (2025)
- [A Survey on the Role of Artificial Intelligence and Machine Learning in 6G-V2X Applications](https://mlsanity.com/paper/2506.09512/) (2025)
- [VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion](https://mlsanity.com/paper/2512.02844/) (2025)
- [Risk-Controllable Multi-View Diffusion for Driving Scenario Generation](https://mlsanity.com/paper/2603.11534/) (2026)
- [Driver Assistant: Persuading Drivers to Adjust Secondary Tasks Using Large Language Models](https://mlsanity.com/paper/2508.05238/) (2025)
- [Downstream-Aware Context Selection for Online In-Context Reinforcement Learning](https://mlsanity.com/paper/2609.33166/) (2026)

## Abstract (arXiv)

Recent advancements in autonomous vehicles (AVs) use Large Language Models (LLMs) to perform well in normal driving scenarios. However, ensuring safety in dynamic, high-risk environments and managing safety-critical long-tail events remain significant challenges. To address these issues, we propose SafeDrive, a knowledge- and data-driven risk-sensitive decision-making framework to enhance AV safety and adaptability. The proposed framework introduces a modular system comprising: (1) a Risk Module for quantifying multi-factor coupled risks involving driver, vehicle, and road interactions; (2) a Memory Module for storing and retrieving typical scenarios to improve adaptability; (3) a LLM-powered Reasoning Module for context-aware safety decision-making; and (4) a Reflection Module for refining decisions through iterative learning. By integrating knowledge-driven insights with adaptive learning mechanisms, the framework ensures robust decision-making under uncertain conditions. Extensive evaluations on real-world traffic datasets, including highways (HighD), intersections (InD), and roundabouts (RounD), validate the framework's ability to enhance decision-making safety (achieving a 100% safety rate), replicate human-like driving behaviors (with decision alignment exceeding 85%), and adapt effectively to unpredictable scenarios. SafeDrive establishes a novel paradigm for integrating knowledge- and data-driven methods, highlighting significant potential to improve safety and adaptability of autonomous driving in high-risk traffic scenarios. Project Page: https://mezzi33.github.io/SafeDrive/