mlsanityData to 7 Oct 2026

LMDrive: Closed-Loop End-to-End Driving with Large Language Models

arXiv 2312.07488 ↗First posted 12 Dec 2023cs.CV cs.AI cs.ROcited by 301

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes, models, and datasets can be found at https://github.com/opendilab/LMDrive

Key facts. Cited by 301 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 47 of its 56 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

47 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  2. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2016Layer Normalizationcited by 5,579
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
32 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  4. 2018SlowFast Networks for Video Recognitioncited by 1,862
  5. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  6. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  7. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  8. 2022Planning-oriented Autonomous Drivingcited by 1,068
  9. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  12. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  13. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  14. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  15. 2019Learning by Cheatingcited by 361
  16. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  17. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  18. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  19. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  20. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  21. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  22. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  23. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  24. 2022Learning from All Vehiclescited by 135
  25. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  26. 2022TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarioscited by 104
  27. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  28. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  29. 2023ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modelingcited by 60
  30. 2020Temporal Interlacing Networkcited by 45
  31. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priorscited by 36
  32. 2022Accelerating Reinforcement Learning for Autonomous Driving using Task-Agnostic and Ego-Centric Motion Skillscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 62; 2025: 109; 2026: 128.

301 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  2. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  3. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  4. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  5. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  6. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  7. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  8. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  9. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  10. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  11. 2024Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodologycited by 92
  12. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
38 more
  1. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
  2. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  3. 2024CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLMcited by 74
  4. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  5. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  6. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  7. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  8. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  9. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  10. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  11. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  12. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  13. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  14. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  15. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  16. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  17. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  18. 2025VisRL: Intention-Driven Visual Perception via Reinforced Reasoningcited by 39
  19. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  20. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  21. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  22. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  23. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  24. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  25. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  26. 2025HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decodercited by 35
  27. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  28. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34
  29. 2024SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Predictioncited by 33
  30. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  31. 2025AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Drivingcited by 32
  32. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  33. 2024Language-Image Models with 3D Understandingcited by 31
  34. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  35. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  36. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  37. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  38. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27

50 most cited of 301 All 301 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
  2. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  3. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
  4. 2024XLM for Autonomous Driving Systems: A Comprehensive Review
  5. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
  6. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs
  7. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  8. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models