mlsanityData to 14 Oct 2026

DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models

arXiv 2309.16292 ↗First posted 28 Sep 2023cs.RO cs.CLcited by 207

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Recent advancements in autonomous driving have relied on data-driven approaches, which are widely adopted but face challenges including dataset bias, overfitting, and uninterpretability. Drawing inspiration from the knowledge-driven nature of human driving, we explore the question of how to instill similar capabilities into autonomous driving systems and summarize a paradigm that integrates an interactive environment, a driver agent, as well as a memory component to address this question. Leveraging large language models (LLMs) with emergent abilities, we propose the DiLu framework, which combines a Reasoning and a Reflection module to enable the system to perform decision-making based on common-sense knowledge and evolve continuously. Extensive experiments prove DiLu's capability to accumulate experience and demonstrate a significant advantage in generalization ability over reinforcement learning-based methods. Moreover, DiLu is able to directly acquire experiences from real-world datasets which highlights its potential to be deployed on practical autonomous driving systems. To the best of our knowledge, we are the first to leverage knowledge-driven capability in decision-making for autonomous vehicles. Through the proposed DiLu framework, LLM is strengthened to apply knowledge and to reason causally in the autonomous driving domain. Project page: https://pjlab-adg.github.io/DiLu/

Key facts. Cited by 207 papers in the map (data through 14 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 34 of its 45 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

34 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
21 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2017Billion-scale similarity search with GPUscited by 2,497
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  7. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  8. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  9. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 289
  10. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  11. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  12. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  13. 2023Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveyscited by 75
  14. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  15. 2022CitySim: A Drone-Based Vehicle Trajectory Dataset for Safety Oriented Research and Digital Twinscited by 43
  16. 2019Towards Corner Case Detection for Autonomous Drivingcited by 39
  17. 2021Description of Corner Cases in Automated Driving: Goals and Challengescited by 35
  18. 2021An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Drivingcited by 33
  19. 2023Milestones in Autonomous Driving and Intelligent Vehicles Part I: Control, Computing System Design, Communication, HD Map, Testing, and Human Behaviorscited by 25
  20. 2023Milestones in Autonomous Driving and Intelligent Vehicles Part II: Perception and Planningcited by 9
  21. 2023Bringing Diversity to Autonomous Vehicles: An Interpretable Multi-vehicle Decision-making and Planning Frameworkcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 72; 2025: 75; 2026: 44.

207 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  3. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
  4. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  5. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  6. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  7. 2023A Language Agent for Autonomous Drivingcited by 164
  8. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Spacecited by 155
  9. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 150
  10. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  11. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 132
  12. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
38 more
  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  2. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
  3. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  4. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  5. 2025MemEvolve: Meta-Evolution of Agent Memory Systemscited by 95
  6. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 80
  7. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 77
  8. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  9. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  10. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  11. 2024A Survey for Foundation Models in Autonomous Drivingcited by 52
  12. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 52
  13. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 51
  14. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 48
  15. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  16. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  17. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 41
  18. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  19. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  20. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  21. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  22. 2024DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenescited by 35
  23. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  24. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  25. 2024ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObilitycited by 34
  26. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 33
  27. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 33
  28. 2024Language-Image Models with 3D Understandingcited by 31
  29. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  30. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  31. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 27
  32. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  33. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  34. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 25
  35. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  36. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  37. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  38. 2024AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Mergingcited by 22

50 most cited of 207 All 207 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles
  2. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model
  3. 2025AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving
  4. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models
  5. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learning
  6. 2026Large Language Model based Interactive Decision-Making for Autonomous Driving
  7. 2026Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges
  8. 2021A Versatile and Efficient Reinforcement Learning Framework for Autonomous Driving