mlsanityData to 7 Oct 2026

Empowering Autonomous Driving with Large Language Models: A Safety Perspective

arXiv 2312.00812 ↗First posted 28 Nov 2023cs.AI cs.LG cs.SYcited by 34

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Autonomous Driving (AD) encounters significant safety hurdles in long-tail unforeseen driving scenarios, largely stemming from the non-interpretability and poor generalization of the deep neural networks within the AD system, particularly in out-of-distribution and uncertain data. To this end, this paper explores the integration of Large Language Models (LLMs) into AD systems, leveraging their robust common-sense knowledge and reasoning abilities. The proposed methodologies employ LLMs as intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning, for enhancing driving performance and safety. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine. Demonstrating superior performance and safety metrics compared to state-of-the-art approaches, our approach shows the promising potential for using LLMs for autonomous vehicles.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 45 of its 56 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

45 of 56 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving2 Nov
  2. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  4. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  5. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles19 Sep
  6. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  8. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  9. 2019Control Barrier Functions: Theory and Applicationscited by 332
  10. 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 313
  11. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  12. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
27 more
  1. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  2. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  3. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 110
  4. 2023Language Prompt for Autonomous Drivingcited by 110
  5. 2022A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspectivecited by 100
  6. 2021Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functionscited by 91
  7. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  8. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  9. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  10. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  11. 2023TrafficGPT: Viewing, Processing and Interacting with Traffic Foundation Modelscited by 45
  12. 2021POLAR: A Polynomial Arithmetic Framework for Verifying Neural-Network Controlled Systemscited by 38
  13. 2023MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Unsignalized Intersectionscited by 38
  14. 2020Neural Certificates for Safe Control Policiescited by 35
  15. 2023Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous drivingcited by 30
  16. 2021Verification of Neural-Network Control Systems by Integrating Taylor Models and Zonotopescited by 26
  17. 2022Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopescited by 23
  18. 2022Semi-supervised Semantics-guided Adversarial Training for Trajectory Predictioncited by 12
  19. 2023POLAR-Express: Efficient and Precise Formal Reachability Analysis of Neural-Network Controlled Systemscited by 12
  20. 2022Joint Differentiable Optimization and Verification for Certified Reinforcement Learningcited by 11
  21. 2022TAE: A Semi-supervised Controllable Behavior-aware Trajectory Generator and Predictorcited by 10
  22. 2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
  23. 2022Differentiable Safe Controller Design through Control Barrier Functionscited by 8
  24. 2020Energy-Efficient Control Adaptation with Safety Guarantees for Learning-Enabled Cyber-Physical Systemscited by 7
  25. 2023Learning Representation for Anomaly Detection of Vehicle Trajectoriescited by 6
  26. 2023Kinematics-aware Trajectory Generation and Prediction with Latent Stochastic Differential Modelingcited by 5
  27. 2023Safety-Assured Speculative Planning with Adaptive Predictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 15; 2025: 11; 2026: 6.

34 papers.

Sort:

By area:

  1. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  2. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  3. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  4. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  5. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  6. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  7. 2024VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenescited by 22
  8. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15
  9. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14
  10. 2024A Survey on Human-AI Collaboration with Large Foundation Modelscited by 7
  11. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  12. 2024Exploring the Causality of End-to-End Autonomous Drivingcited by 7
22 more
  1. 2024Enhancing LLM-based Autonomous Driving Agents to Mitigate Perception Attackscited by 6
  2. 2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
  3. 2026HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defensecited by 6
  4. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  5. 2025Planning Safety Trajectories with Dual-Phase, Physics-Informed, and Transportation Knowledge-Driven Large Language Modelscited by 5
  6. 2025LogicGuard: Improving Embodied LLM agents through Temporal Logic based Criticscited by 5
  7. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 3
  8. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  9. 2024Leveraging Edge Intelligence and LLMs to Advance 6G-Enabled Internet of Automated Defense Vehiclescited by 3
  10. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  11. 2025DriveMind: A Dual Visual Language Model-based Reinforcement Learning Framework for Autonomous Drivingcited by 3
  12. 2025Large Language Model-assisted Autonomous Vehicle Recovery from Immobilizationcited by 3
  13. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  14. 2025LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecturecited by 2
  15. 2025Your Ride, Your Rules: Psychology and Cognition Enabled Automated Driving Systemscited by 1
  16. 2025Last Layer Hamiltonian Monte Carlocited by 1
  17. 2026STEP-LLM: Generating CAD STEP Models from Natural Language with Large Language Modelscited by 1
  18. 2026VL-DPO: Vision-Language-Guided Finetuning for Preference-Aligned Autonomous Drivingcited by 1
  19. 2025HySafe-AI: Hybrid Safety Architectural Analysis Framework for AI Systems: A Case Studycited by 0
  20. 2026Frozen LLMs as Map-Aware Spatio-Temporal Reasoners for Vehicle Trajectory Predictioncited by 0
  21. 2026GUT: Quantifying and Optimizing the Reasoning Uncertainty of LLMs via Graph Complexitycited by 0
  22. 2026Large Language Models in the Loop: A Stability- and Network-Aware Survey in Networked Control, Cyber-Physical, and Multi-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Driving Through Uncertainty: Risk-Averse Control with LLM Commonsense for Autonomous Driving under Perception Deficits
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  3. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
  4. 2025Enhancing Autonomous Driving Systems with On-Board Deployed Large Language Models
  5. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
  6. 2026SARAD: LLM-Based Safety-Aware Hybrid Reinforcement Learning with Collision Prediction for Autonomous Driving
  7. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  8. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving