mlsanityData to 7 Oct 2026

Personalized Autonomous Driving with Large Language Models: Field Experiments

arXiv 2312.09397 ↗First posted 14 Dec 2023cs.AIcited by 46

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Integrating large language models (LLMs) in autonomous vehicles enables conversation with AI systems to drive the vehicle. However, it also emphasizes the requirement for such systems to comprehend commands accurately and achieve higher-level personalization to adapt to the preferences of drivers or passengers over a more extended period. In this paper, we introduce an LLM-based framework, Talk2Drive, capable of translating natural verbal commands into executable controls and learning to satisfy personal preferences for safety, efficiency, and comfort with a proposed memory module. This is the first-of-its-kind multi-scenario field experiment that deploys LLMs on a real-world autonomous vehicle. Experiments showcase that the proposed system can comprehend human intentions at different intuition levels, ranging from direct commands like "can you drive faster" to indirect commands like "I am really in a hurry now". Additionally, we use the takeover rate to quantify the trust of human drivers in the LLM-based autonomous driving system, where Talk2Drive significantly reduces the takeover rate in highway, intersection, and parking scenarios. We also validate that the proposed memory module considers personalized preferences and further reduces the takeover rate by up to 65.2% compared with those without a memory module. The experiment video can be watched at https://www.youtube.com/watch?v=4BWsfPaq1Ro

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 15 of its 32 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

15 of 32 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs7 Dec
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Driving21 Nov
  3. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles12 Oct
  4. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  6. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  7. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles19 Sep
  8. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 7

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  6. 2022Driver Digital Twin for Online Prediction of Personalized Lane Change Behaviorcited by 13
  7. 2021EfficientWord-Net: An Open Source Hotword Detection Engine based on One-shot Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 13; 2026: 22.

46 papers.

Sort:

By area:

  1. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  2. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 127
  3. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 113
  4. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  5. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  6. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  7. 2025StyleDrive: Towards Driving-Style Aware Benchmarking of End-To-End Autonomous Drivingcited by 20
  8. 2024Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typographycited by 17
  9. 2024Towards Interactive and Learnable Cooperative Driving Automation: a Large Language Model-Driven Decision-Making Frameworkcited by 16
  10. 2024On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validationcited by 15
  11. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigationcited by 14
  12. 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agentscited by 12
34 more
  1. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  2. 2024Large Language Model guided Deep Reinforcement Learning for Decision Making in Autonomous Drivingcited by 8
  3. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  4. 2024From Words to Wheels: Automated Style-Customized Policy Generation for Autonomous Drivingcited by 5
  5. 2025A Comprehensive LLM-powered Framework for Driving Intelligence Evaluationcited by 5
  6. 2026Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Drivingcited by 4
  7. 2026FragFuse: Bypassing Access Control of Large Language Model Agents via Memory-Based Query Fragmentation and Fusioncited by 4
  8. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 3
  9. 2025Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactionscited by 3
  10. 2026ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Drivingcited by 3
  11. 2025Sage Deer: A Super-Aligned Driving Generalist Is Your Copilotcited by 2
  12. 2025FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systemscited by 2
  13. 2026Can Users Specify Driving Speed? Bench2Drive-Speed: Benchmark and Baselines for Desired-Speed Conditioned Autonomous Drivingcited by 2
  14. 2025Box-QAymo: Box-Referring VQA Dataset for Autonomous Drivingcited by 1
  15. 2026Modular Autonomy with Conversational Interaction: An LLM-driven Framework for Decision Making in Autonomous Drivingcited by 1
  16. 2026Driving Like Yourself: A Benchmark for Closed-Loop Personalized End-to-End Autonomous Drivingcited by 1
  17. 2026Efficient and Explainable End-to-End Autonomous Driving via Masked Vision-Language-Action Diffusioncited by 1
  18. 2026MILD: Mediator Agent System with Bidirectional Perception and Multi-Layered Alignment for Human-Vehicle Collaborationcited by 1
  19. 2025Scene-Aware Conversational ADAS with Generative AI for Real-Time Driver Assistancecited by 0
  20. 2025Adaptive Content Restriction for Large Language Models via Suffix Optimizationcited by 0
  21. 2025BarrierBench: Evaluating Large Language Models for Safety Verification in Dynamical Systemscited by 0
  22. 2026CARE Drive A Framework for Evaluating Reason-Responsiveness of Vision Language Models in Automated Drivingcited by 0
  23. 2026Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous Vehiclescited by 0
  24. 2026Information Coordination as a Bridge: A Neuro-Symbolic Architecture for Reliable Autonomous Driving Scene Understandingcited by 0
  25. 2026Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Drivingcited by 0
  26. 2026I'm Sorry Driver, I'm Afraid I Can't Do That: Appraising the Safety of LLMs within Automotive Contextscited by 0
  27. 2026An LLM-Explainable DRL Framework for Passenger-Directed Autonomous Drivingcited by 0
  28. 2026Safety-Aware Evaluation of LLM-Generated Driver Intervention Messages through Multi-Task Risk Fusioncited by 0
  29. 2026VISTA-DZ: Visual Semantic Trajectory Adaptation for Personalized Dilemma Zone Predictioncited by 0
  30. 2026A Large-Language-Model Supported Personalized Driving Framework for Lane Change in Highway Scenarioscited by 0
  31. 2026Beyond Scene Priors: Fine-Grained Traffic Scene Reasoning with Benchmarking and Query-Guided Small-Object Focuscited by 0
  32. 2026Query-Only Backdoor Attacks on Self-Evolving Skills via Trajectory Poisoningcited by 0
  33. 2026From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Callscited by 0
  34. 2026DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving
  2. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  3. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  4. 2023A Language Agent for Autonomous Driving
  5. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  6. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoning
  7. 2025AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving
  8. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving