mlsanityData to 7 Oct 2026

DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model

arXiv 2310.01412 ↗First posted 2 Oct 2023cs.CV cs.ROcited by 455

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) have emerged as a prominent area of interest within the research community, given their proficiency in handling and reasoning with non-textual data, including images and videos. This study seeks to extend the application of MLLMs to the realm of autonomous driving by introducing DriveGPT4, a novel interpretable end-to-end autonomous driving system based on LLMs. Capable of processing multi-frame video inputs and textual queries, DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users. Furthermore, DriveGPT4 predicts low-level vehicle control signals in an end-to-end fashion.These advanced capabilities are achieved through the utilization of a bespoke visual instruction tuning dataset, specifically tailored for autonomous driving applications, in conjunction with a mix-finetuning training strategy. DriveGPT4 represents the pioneering effort to leverage LLMs for the development of an interpretable end-to-end autonomous driving solution. Evaluations conducted on the BDD-X dataset showcase the superior qualitative and quantitative performance of DriveGPT4. Additionally, the fine-tuning of domain-specific data enables DriveGPT4 to yield close or even improved results in terms of autonomous driving grounding when contrasted with GPT4-V.

Key facts. Cited by 455 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 63 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

44 of 63 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
32 more
  1. 2016Pyramid Scene Parsing Networkcited by 2,472
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2022Planning-oriented Autonomous Drivingcited by 1,068
  10. 2016End to End Learning for Self-Driving Carscited by 1,000
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  12. 2023Instruction Tuning with GPT-4cited by 676
  13. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  14. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  15. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  16. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  17. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  18. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  19. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  20. 2021Translating Images into Mapscited by 115
  21. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  22. 2023Language Prompt for Autonomous Drivingcited by 110
  23. 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
  24. 2023Real-Time Flying Object Detection with YOLOv8cited by 93
  25. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  26. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  27. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  28. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  29. 2018Learning Attraction Field Representation for Robust Line Segment Detectioncited by 52
  30. 2021Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillationcited by 25
  31. 2022RNGDet++: Road Network Graph Detection by Transformer with Instance Segmentation and Multi-scale Features Enhancementcited by 24
  32. 2022CenterLineDet: CenterLine Graph Detection for Road Lanes with Vehicle-mounted Sensors by Transformer for HD Map Generationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 122; 2025: 187; 2026: 125.

455 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  3. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  4. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  5. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  6. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  7. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  8. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  9. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  10. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  11. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  12. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
38 more
  1. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  3. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  4. 2023A Language Agent for Autonomous Drivingcited by 161
  5. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  6. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  7. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  8. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  9. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  10. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  11. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  12. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  13. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  14. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  15. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  16. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  17. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  18. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  19. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  20. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  21. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  22. 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehiclescited by 92
  23. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  24. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  25. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  26. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  27. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  28. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  29. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  30. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  31. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 74
  32. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  33. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  34. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  35. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  36. 2024Embodied Understanding of Driving Scenarioscited by 63
  37. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  38. 2025WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarioscited by 60

50 most cited of 455 All 455 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
  2. 2024Probing Multimodal LLMs as World Models for Driving
  3. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  5. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
  6. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models
  7. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
  8. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving