mlsanityData to 7 Oct 2026

Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles

arXiv 2310.08034 ↗First posted 12 Oct 2023cs.HC cs.AI cs.ROcited by 72

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 1 more
Abstract (arXiv)

The fusion of human-centric design and artificial intelligence (AI) capabilities has opened up new possibilities for next-generation autonomous vehicles that go beyond transportation. These vehicles can dynamically interact with passengers and adapt to their preferences. This paper proposes a novel framework that leverages Large Language Models (LLMs) to enhance the decision-making process in autonomous vehicles. By utilizing LLMs' linguistic and contextual understanding abilities with specialized tools, we aim to integrate the language and reasoning capabilities of LLMs into autonomous vehicles. Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks, to explore LLMs' interpretation, interaction, and reasoning in various scenarios. We also examine real-time personalization, demonstrating how LLMs can influence driving behaviors based on verbal commands. Our empirical results highlight the substantial advantages of utilizing chain-of-thought prompting, leading to improved driving decisions, and showing the potential for LLMs to enhance personalized driving experiences through ongoing verbal feedback. The proposed framework aims to transform autonomous vehicle operations, offering personalized support, transparent decision-making, and continuous learning to enhance safety and effectiveness. We achieve user-centric, transparent, and adaptive autonomous driving ecosystems supported by the integration of LLMs into autonomous vehicles.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 32 of its 61 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

32 of 61 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
20 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  7. 2022PAL: Program-aided Language Modelscited by 635
  8. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  9. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  10. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  11. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  12. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  13. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  14. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  15. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  16. 2023TrafficSafetyGPT: Tuning a Pre-trained Large Language Model to a Domain-Specific Expert in Transportation Safetycited by 27
  17. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  18. 2023Mitigating Transformer Overconfidence via Lipschitz Regularizationcited by 15
  19. 2022THMA: Tencent HD Map AI System for Creating HD Map Annotationscited by 9
  20. 2023Radar Enlighten the Dark: Enhancing Low-Visibility Perception for Automated Vehicles with Camera-Radar Fusioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 26; 2025: 27; 2026: 11.

72 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  4. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  5. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  6. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  7. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  8. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 70
  9. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  10. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 46
  11. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  12. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
38 more
  1. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  2. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  3. 2024LLM-Assisted Light: Leveraging Large Language Model Capabilities for Human-Mimetic Traffic Signal Control in Complex Urban Environmentscited by 31
  4. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  5. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  6. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22
  7. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  8. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  9. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  10. 2024Probing Multimodal LLMs as World Models for Drivingcited by 19
  11. 2024Large Action Models: From Inception to Implementationcited by 18
  12. 2025RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Drivingcited by 18
  13. 2024On the Brittle Foundations of ReAct Prompting for Agentic Large Language Modelscited by 17
  14. 2025DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Modelscited by 17
  15. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Modelscited by 16
  16. 2024Towards Interactive and Learnable Cooperative Driving Automation: a Large Language Model-Driven Decision-Making Frameworkcited by 16
  17. 2024Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Gamecited by 12
  18. 2024Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigmcited by 12
  19. 2024Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testingcited by 10
  20. 2024SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Modelscited by 10
  21. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  22. 2025DriveQA: Passing the Driving Knowledge Testcited by 10
  23. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  24. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  25. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  26. 2025DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Modelscited by 7
  27. 2025CoopReflect: Towards Natural Language Communication for Cooperative Autonomous Driving via Multi-Agent Learningcited by 7
  28. 2024Advancing Explainable Autonomous Vehicle Systems: A Comprehensive Review and Research Roadmapcited by 6
  29. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  30. 2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
  31. 2024From Words to Wheels: Automated Style-Customized Policy Generation for Autonomous Drivingcited by 5
  32. 2024TransitGPT: A Generative AI-based framework for interacting with GTFS data using Large Language Modelscited by 5
  33. 2025A Cascading Cooperative Multi-agent Framework for On-ramp Merging Control Integrating Large Language Modelscited by 5
  34. 2025PADriver: Towards Personalized Autonomous Drivingcited by 5
  35. 2025"I Can See Forever!": Evaluating Real-time VideoLLMs for Assisting Individuals with Visual Impairmentscited by 3
  36. 2025Large Language Model-assisted Autonomous Vehicle Recovery from Immobilizationcited by 3
  37. 2025CoordField: Coordination Field for Agentic UAV Task Allocation In Low-altitude Urban Scenarioscited by 2
  38. 2025LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecturecited by 2

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving
  2. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  3. 2023Evaluation of Large Language Models for Decision Making in Autonomous Driving
  4. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models
  5. 2024Leveraging Large Language Models for Enhancing Autonomous Vehicle Perception
  6. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  7. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  8. 2023A Language Agent for Autonomous Driving