mlsanityData to 7 Oct 2026

LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

arXiv 2310.03026 ↗First posted 4 Oct 2023cs.RO cs.AI cs.CLcited by 188

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Existing learning-based autonomous driving (AD) systems face challenges in comprehending high-level information, generalizing to rare events, and providing interpretability. To address these problems, this work employs Large Language Models (LLMs) as a decision-making component for complex AD scenarios that require human commonsense understanding. We devise cognitive pathways to enable comprehensive reasoning with LLMs, and develop algorithms for translating LLM decisions into actionable driving commands. Through this approach, LLM decisions are seamlessly integrated with low-level controllers by guided parameter matrix adaptation. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination, thanks to the commonsense reasoning capabilities of LLMs. This paper presents an initial step toward leveraging LLMs as effective decision-makers for intricate AD scenarios in terms of safety, efficiency, generalizability, and interoperability. We aspire for it to serve as inspiration for future research in this field. Project page: https://sites.google.com/view/llm-mpc

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 72 of its 91 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

72 of 91 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  3. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  4. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles19 Sep
  5. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
38 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  9. 2022Planning-oriented Autonomous Drivingcited by 1,068
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  12. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  13. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  14. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  15. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  16. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  17. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  18. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  19. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  20. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  21. 2023Reward Design with Language Modelscited by 257
  22. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  23. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  24. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  25. 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 128
  26. 2020A Survey of End-to-End Driving: Architectures and Training Methodscited by 123
  27. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  28. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  29. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  30. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  31. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  32. 2023LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusioncited by 80
  33. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  34. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  35. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  36. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  37. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  38. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 79; 2025: 62; 2026: 27.

188 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  3. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  4. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  5. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  6. 2023A Language Agent for Autonomous Drivingcited by 161
  7. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  8. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  9. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  10. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  11. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  12. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
38 more
  1. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  2. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  3. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  4. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  5. 2023LLMLight: Large Language Models as Traffic Signal Control Agentscited by 73
  6. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 68
  7. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  8. 2024Embodied Understanding of Driving Scenarioscited by 63
  9. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  10. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  11. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  12. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  13. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  14. 2024MoralBench: Moral Evaluation of LLMscited by 50
  15. 2024VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Modelscited by 46
  16. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  17. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  18. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Followingcited by 41
  19. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  20. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  21. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  22. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  23. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  24. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  25. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  26. 2024ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObilitycited by 34
  27. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34
  28. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  29. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
  30. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotationscited by 31
  31. 2023Large Language Model as a Policy Teacher for Training Reinforcement Learning Agentscited by 30
  32. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  33. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  34. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  35. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  36. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  37. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  38. 2024EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agentscited by 24

50 most cited of 188 All 188 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
  2. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  3. 2025ADRD: LLM-Driven Autonomous Driving Based on Rule-based Decision Systems
  4. 2023Personalized Autonomous Driving with Large Language Models: Field Experiments
  5. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Models
  6. 2025Enhancing Autonomous Driving Systems with On-Board Deployed Large Language Models
  7. 2026Large Language Model based Interactive Decision-Making for Autonomous Driving
  8. 2026Multi-Agent Orchestration with the Common-Sense Reasoning Capabilities of LLMs for Autonomous Driving