mlsanityData to 7 Oct 2026

LLM4Drive: A Survey of Large Language Models for Autonomous Driving

arXiv 2311.01043 ↗First posted 2 Nov 2023cs.AIcited by 136

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Autonomous driving technology, a catalyst for revolutionizing transportation and urban mobility, has the tend to transition from rule-based systems to data-driven strategies. Traditional module-based systems are constrained by cumulative errors among cascaded modules and inflexible pre-set rules. In contrast, end-to-end autonomous driving systems have the potential to avoid error accumulation due to their fully data-driven training process, although they often lack transparency due to their "black box" nature, complicating the validation and traceability of decisions. Recently, large language models (LLMs) have demonstrated abilities including understanding context, logical reasoning, and generating answers. A natural thought is to utilize these abilities to empower autonomous driving. By combining LLM with foundation vision models, it could open the door to open-world understanding, reasoning, and few-shot learning, which current autonomous driving systems are lacking. In this paper, we systematically review a research line about textitLarge Language Models for Autonomous Driving (LLM4AD). This study evaluates the current state of technological advancements, distinctly outlining the principal challenges and prospective directions for the field. For the convenience of researchers in academia and industry, we provide real-time updates on the latest advances in the field as well as relevant open-source resources via the designated link: https://github.com/Thinklab-SJTU/Awesome-LLM4AD.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 85 of its 166 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

85 of 166 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles12 Oct
  2. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  3. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  5. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  6. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles19 Sep
  7. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 78

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2022Training language models to follow instructions with human feedbackcited by 15,712
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Generative Adversarial Networkscited by 5,603
  12. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2015Variational Inference with Normalizing Flowscited by 2,161
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  8. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  9. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  10. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  11. 2022Planning-oriented Autonomous Drivingcited by 1,068
  12. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  13. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  14. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  15. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  16. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  17. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  18. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  19. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  20. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  21. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  22. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  23. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  24. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  25. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  26. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  27. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  28. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  29. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  30. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  31. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  32. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  33. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  34. 2021Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Trackingcited by 165
  35. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  36. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  37. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  38. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 42; 2025: 66; 2026: 24.

136 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  3. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  4. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  6. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  8. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  9. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  10. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  11. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  12. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
38 more
  1. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  2. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)cited by 51
  3. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  4. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  5. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  6. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  7. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  8. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  9. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  10. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  11. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  12. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  13. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  14. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insightscited by 30
  15. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  16. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  17. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  18. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  19. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  20. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  21. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  22. 2024SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Modelscited by 21
  23. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  24. 2025When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysiscited by 20
  25. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  26. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  27. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  28. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  29. 2024Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typographycited by 17
  30. 2024ActiveAD: Planning-Oriented Active Learning for End-to-End Autonomous Drivingcited by 16
  31. 2025LLM-attacker: Enhancing Closed-loop Adversarial Scenario Generation for Autonomous Driving with Large Language Modelscited by 16
  32. 2025Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Drivingcited by 15
  33. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  34. 2025V2X-UniPool: Unifying Multimodal Perception and Knowledge Reasoning for Autonomous Drivingcited by 14
  35. 2025TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learningcited by 13
  36. 2025Trajectory Prediction Meets Large Language Models: A Surveycited by 13
  37. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  38. 2025PSG-Agent: Personality-Aware Safety Guardrail for LLM-based Agentscited by 12

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  2. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
  3. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  4. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  5. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  6. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  7. 2024A Survey on Large Language Model-empowered Autonomous Driving
  8. 2023Evaluation of Large Language Models for Decision Making in Autonomous Driving