mlsanityData to 7 Oct 2026

A Survey on Large Language Model-empowered Autonomous Driving

arXiv 2409.14165 ↗First posted 21 Sep 2024cs.AI cs.CL cs.LGcited by 3

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Artificial intelligence (AI) plays a crucial role in autonomous driving (AD) research, propelling its development towards intelligence and efficiency. Currently, the development of AD technology follows two main technical paths: modularization and end-to-end. Modularization decompose the driving task into modules such as perception, prediction, planning, and control, and train them separately. Due to the inconsistency of training objectives between modules, the integrated effect suffers from bias. End-to-end attempts to address this issue by utilizing a single model that directly maps from sensor data to control signals. This path has limited learning capabilities in a comprehensive set of features and struggles to handle unpredictable long-tail events and complex urban traffic scenarios. In the face of challenges encountered in both paths, many researchers believe that large language models (LLMs) with powerful reasoning capabilities and extensive knowledge understanding may be the solution, expecting LLMs to provide AD systems with deeper levels of understanding and decision-making capabilities. In light of the challenges faced by both paths, many researchers believe that LLMs, with their powerful reasoning abilities and extensive knowledge, could offer a solution. To understand if LLMs could enhance AD, this paper conducts a thorough analysis of the potential applications of LLMs in AD systems, including exploring their optimization strategies in both modular and end-to-end approaches, with a particular focus on how LLMs can tackle the problems and challenges present in current solutions. Furthermore, we discuss an important question: Can LLM-based artificial general intelligence (AGI) be a key to achieve high-level AD? We further analyze the potential limitations and challenges that LLMs may encounter in promoting the development of AD technology.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 78 of its 115 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

78 of 115 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving21 Feb
  2. 2024Driving Everywhere with Large Language Model Policy Adaptation8 Feb
  3. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Driving2 Feb
  4. 2024LangProp: A code optimization framework using Large Language Models applied to driving18 Jan
  5. 2023Personalized Autonomous Driving with Large Language Models: Field Experiments14 Dec
  6. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  7. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Models12 Dec
  8. 2023Evaluation of Large Language Models for Decision Making in Autonomous Driving11 Dec
  9. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs7 Dec
  10. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspective28 Nov
  11. 2023A Language Agent for Autonomous Driving17 Nov
  12. 2023Dialogue-based generation of self-driving simulation scenarios using Large Language Models26 Oct
6 more
  1. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  2. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  4. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  5. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles19 Sep
  6. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  5. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  6. 2016End to End Learning for Self-Driving Carscited by 1,000
  7. 2023Large Language Modelscited by 774
  8. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  10. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  11. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  12. 2019Learning by Cheatingcited by 361
38 more
  1. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  2. 2018Learning to Drive in a Daycited by 266
  3. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  4. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  5. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  6. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  7. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  8. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  9. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  10. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  11. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  12. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  13. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  14. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  15. 2023Language Conditioned Traffic Generationcited by 86
  16. 2019Urban Driving with Conditional Imitation Learningcited by 84
  17. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  18. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  19. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  20. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  21. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  22. 2023LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoningcited by 59
  23. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  24. 2023Semantic Anomaly Detection with Large Language Modelscited by 53
  25. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  26. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 41
  27. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  28. 2019Deep Learning-based Vehicle Behaviour Prediction For Autonomous Driving Applications: A Reviewcited by 31
  29. 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Modelscited by 31
  30. 2022HybridNets: End-to-End Perception Networkcited by 30
  31. 2023Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous drivingcited by 30
  32. 2023WEDGE: A multi-weather autonomous driving dataset built from generative vision-language modelscited by 28
  33. 2023Milestones in Autonomous Driving and Intelligent Vehicles Part I: Control, Computing System Design, Communication, HD Map, Testing, and Human Behaviorscited by 25
  34. 2023The Moral Machine Experiment on Large Language Modelscited by 24
  35. 2023SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Datacited by 24
  36. 2024LG-Traj: LLM Guided Pedestrian Trajectory Predictioncited by 24
  37. 2023Large Trajectory Models are Scalable Motion Predictors and Plannerscited by 23
  38. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  2. 2025V2X-UniPool: Unifying Multimodal Perception and Knowledge Reasoning for Autonomous Drivingcited by 14
  3. 2025SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Drivingcited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  2. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  3. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
  4. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  5. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances
  6. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
  7. 2023Towards Knowledge-driven Autonomous Driving
  8. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignment