mlsanityData to 14 Oct 2026

LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends

arXiv 2410.15281 ↗First posted 20 Oct 2024cs.RO cs.AI cs.CLcited by 10

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

With the broader adoption and highly successful development of Large Language Models (LLMs), there has been growing interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning capabilities, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to interactive decision-making. This paper first introduces the novel concept of designing Large Language Models for Autonomous Driving (LLM4AD), followed by a review of existing LLM4AD studies. Then, a comprehensive benchmark is proposed for evaluating the instruction-following and reasoning abilities of LLM4AD systems, which includes LaMPilot-Bench, CARLA Leaderboard 1.0 Benchmark in simulation and NuPlanQA for multi-view visual question answering. Furthermore, extensive real-world experiments are conducted on autonomous vehicle platforms, examining both on-cloud and on-edge LLM deployment for personalized decision-making and motion control. Next, the future trends of integrating language diffusion models into autonomous driving are explored, exemplified by the proposed ViLaD (Vision-Language Diffusion) framework. Finally, the main challenges of LLM4AD are discussed, including latency, deployment, security and privacy, safety, trust and transparency, and personalization.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 92 of its 181 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

92 of 181 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF6 Jun
  2. 2024Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Models7 May
  3. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model16 Feb
  4. 2024Driving Everywhere with Large Language Model Policy Adaptation8 Feb
  5. 2024LangProp: A code optimization framework using Large Language Models applied to driving18 Jan
  6. 2023Personalized Autonomous Driving with Large Language Models: Field Experiments14 Dec
  7. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  8. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Models12 Dec
  9. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs7 Dec
  10. 2023A Language Agent for Autonomous Driving17 Nov
  11. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  12. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
4 more
  1. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  2. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  3. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles19 Sep
  4. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 76

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  10. 2016Deep Learning with Differential Privacycited by 2,728
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  13. 2023A Survey of Large Language Modelscited by 2,576
  14. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  15. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  16. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  17. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  18. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  19. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  20. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  21. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  22. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  23. 2023PaLM 2 Technical Reportcited by 1,162
  24. 2022Planning-oriented Autonomous Drivingcited by 1,087
  25. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  26. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  27. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 537
  28. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  29. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  30. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  31. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  32. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  33. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  34. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
  35. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  36. 2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 287
  37. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  38. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 252

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  2. 2026AgentDrive: An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systemscited by 4
  3. 2026Efficient and Explainable End-to-End Autonomous Driving via Masked Vision-Language-Action Diffusioncited by 2
  4. 2025Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoningcited by 0
  5. 2026SwarmDrive: Semantic V2V Coordination for Latency-Constrained Cooperative Autonomous Drivingcited by 0
  6. 2026Steins;Gate Drive: Semantic Safety Arbitration over Structured Futures for Latency-Decoupled LLM Planningcited by 0
  7. 2026Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Drivingcited by 0
  8. 2026Exploring LLM Capabilities for Situational Understanding and COLREG compliance on real-world maritime navigation scenarioscited by 0
  9. 2026Risk-Adaptive Edge--Cloud Visual Reasoning for Communication-Efficient Autonomous Drivingcited by 0
  10. 2026Talk to Me, Jarvis: An Open-Source Edge-Deployable Voice Assistant Framework for Autonomous Racecarscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  2. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  3. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  4. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
  5. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  6. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
  7. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  8. 2025A Survey on Vision-Language-Action Models for Autonomous Driving