mlsanityData to 14 Oct 2026

Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles

arXiv 2309.10228 ↗First posted 19 Sep 2023cs.HC cs.AIcited by 100

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

The future of autonomous vehicles lies in the convergence of human-centric design and advanced AI capabilities. Autonomous vehicles of the future will not only transport passengers but also interact and adapt to their desires, making the journey comfortable, efficient, and pleasant. In this paper, we present a novel framework that leverages Large Language Models (LLMs) to enhance autonomous vehicles' decision-making processes. By integrating LLMs' natural language capabilities and contextual understanding, specialized tools usage, synergizing reasoning, and acting with various modules on autonomous vehicles, this framework aims to seamlessly integrate the advanced language and reasoning capabilities of LLMs into autonomous vehicles. The proposed framework holds the potential to revolutionize the way autonomous vehicles operate, offering personalized assistance, continuous learning, and transparent decision-making, ultimately contributing to safer and more efficient autonomous driving technologies.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 28 of its 35 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

28 of 35 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
16 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  8. 2022PAL: Program-aided Language Modelscited by 652
  9. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  10. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 289
  11. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  12. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  13. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  14. 2023TrafficSafetyGPT: Tuning a Pre-trained Large Language Model to a Domain-Specific Expert in Transportation Safetycited by 27
  15. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  16. 2023Radar Enlighten the Dark: Enhancing Low-Visibility Perception for Automated Vehicles with Camera-Radar Fusioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 31; 2025: 43; 2026: 9.

100 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  2. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  3. 2023Video Understanding with Large Language Models: A Surveycited by 206
  4. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  5. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  6. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  7. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  8. 2023LLMLight: Large Language Models as Traffic Signal Control Agentscited by 73
  9. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  10. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 52
  11. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49
  12. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 48
38 more
  1. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  2. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  3. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  4. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solvingcited by 38
  5. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  6. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 33
  7. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 32
  8. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  9. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 30
  10. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  11. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 29
  12. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  13. 2023SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Datacited by 24
  14. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  15. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  16. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22
  17. 2024AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Mergingcited by 22
  18. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  19. 2024Probing Multimodal LLMs as World Models for Drivingcited by 20
  20. 2025Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Drivingcited by 18
  21. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Modelscited by 17
  22. 2024Driving Style Alignment for LLM-powered Driver Agentcited by 15
  23. 2024Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Modelscited by 15
  24. 2024Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robotscited by 15
  25. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  26. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14
  27. 2025TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learningcited by 13
  28. 2025Incomplete Multi-view Clustering via Diffusion Contrastive Generationcited by 13
  29. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
  30. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  31. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  32. 2025DriveQA: Passing the Driving Knowledge Testcited by 10
  33. 2025AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Drivingcited by 10
  34. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 10
  35. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  36. 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Drivingcited by 9
  37. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimizationcited by 9
  38. 2024Generating and Evolving Reward Functions for Highway Driving with Large Language Modelscited by 8

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models
  2. 2023A Language Agent for Autonomous Driving
  3. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  4. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  5. 2026Large Language Model based Interactive Decision-Making for Autonomous Driving
  6. 2026Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving
  7. 2025Human-like Semantic Navigation for Autonomous Driving using Knowledge Representation and Large Language Models
  8. 2025A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving