mlsanityData to 7 Oct 2026

DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving

arXiv 2312.09245 ↗First posted 14 Dec 2023cs.CVcited by 233

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have opened up new possibilities for intelligent agents, endowing them with human-like thinking and cognitive abilities. In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD). We introduce DriveMLM, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators. To this end, (1) we bridge the gap between the language decisions and the vehicle control commands by standardizing the decision states according to the off-the-shelf motion planning module. (2) We employ a multimodal LLM (MLLM) to model the behavior planning module of a module AD system, which uses driving rules, user commands, and inputs from various sensors (e.g., camera, lidar) as input and makes driving decisions and provide explanations; This model can plug-and-play in existing AD systems such as Autopilot and Apollo for close-loop driving. (3) We design an effective data engine to collect a dataset that includes decision state and corresponding explanation annotation for model training and evaluation. We conduct extensive experiments and show that replacing the decision-making modules of the Autopilot and Apollo with DriveMLM resulted in significant improvements of 3.2 and 4.7 points on the CARLA Town05 Long respectively, demonstrating the effectiveness of our model. We hope this work can serve as a baseline for autonomous driving with LLMs.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 80 of its 111 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

80 of 111 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  13. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  14. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  16. 2022Planning-oriented Autonomous Drivingcited by 1,068
  17. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  18. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  19. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  20. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  21. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  22. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  23. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  24. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  25. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  26. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  27. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  28. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  29. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  30. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  31. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  32. 2019Learning by Cheatingcited by 361
  33. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  34. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  35. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  36. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  37. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  38. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 75; 2025: 107; 2026: 50.

233 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  2. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  3. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  4. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  5. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  6. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  7. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  8. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  9. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  10. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  11. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  12. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
38 more
  1. 2025Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillationcited by 93
  2. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  3. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  4. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  5. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  6. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  7. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  8. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  9. 2024CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLMcited by 74
  10. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  11. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
  12. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  13. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  14. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 60
  15. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  16. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  17. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  18. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  19. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Drivingcited by 52
  20. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  21. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  22. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  23. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  24. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  25. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  26. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  27. 2025Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Trainingcited by 39
  28. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  29. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  30. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  31. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  32. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  33. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  34. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  35. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
  36. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  37. 2025AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Drivingcited by 32
  38. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32

50 most cited of 233 All 233 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  3. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models
  4. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances
  5. 2025Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving
  6. 2024Probing Multimodal LLMs as World Models for Driving
  7. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
  8. 2026Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving