mlsanityData to 7 Oct 2026

Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving

arXiv 2310.01957 ↗First posted 3 Oct 2023cs.RO cs.AI cs.CLcited by 231

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Large Language Models (LLMs) have shown promise in the autonomous driving sector, particularly in generalization and interpretability. We introduce a unique object-level multimodal LLM architecture that merges vectorized numeric modalities with a pre-trained LLM to improve context understanding in driving situations. We also present a new dataset of 160k QA pairs derived from 10k driving scenarios, paired with high quality control commands collected with RL agent and question answer pairs generated by teacher LLM (GPT-3.5). A distinct pretraining strategy is devised to align numeric vector modalities with static LLM representations using vector captioning language data. We also introduce an evaluation metric for Driving QA and demonstrate our LLM-driver's proficiency in interpreting driving scenarios, answering questions, and decision-making. Our findings highlight the potential of LLM-based driving action generation in comparison to traditional behavioral cloning. We make our benchmark, datasets, and model available for further exploration.

Key facts. Cited by 231 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 44 of its 52 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

44 of 52 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
32 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  6. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2019Attention is not Explanationcited by 916
  10. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  11. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  12. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  13. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  14. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  15. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  16. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  17. 2023GPTScore: Evaluate as You Desirecited by 357
  18. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  19. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  20. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  21. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  22. 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
  23. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  24. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  25. 2021Explanations in Autonomous Driving: A Surveycited by 80
  26. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  27. 2018On Offline Evaluation of Vision-based Driving Modelscited by 62
  28. 2019Conditional Driving from Natural Language Instructionscited by 17
  29. 2021What data do we need for training an AV motion planner?cited by 15
  30. 2021Reimagining an autonomous vehiclecited by 11
  31. 2023Textual Explanations for Automated Commentary Drivingcited by 6
  32. 2018Robot Representation and Reasoning with Knowledge from Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 81; 2025: 97; 2026: 33.

231 papers.

Sort:

By area:

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  2. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  3. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  5. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  6. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  7. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  8. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  9. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  10. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  11. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  12. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
38 more
  1. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  2. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  3. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  4. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  6. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  7. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  8. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  9. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  10. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
  11. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  12. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  13. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  14. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  15. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  16. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  17. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  18. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  19. 2024Embodied Understanding of Driving Scenarioscited by 63
  20. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  21. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  22. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  23. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Drivingcited by 52
  24. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  25. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  26. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 46
  27. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  28. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  29. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  30. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  31. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
  32. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  33. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  34. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  35. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  36. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  37. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understandingcited by 35
  38. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34

50 most cited of 231 All 231 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
  2. 2025Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving
  3. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
  4. 2026Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models
  5. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
  6. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  7. 2026Modular Autonomy with Conversational Interaction: An LLM-driven Framework for Decision Making in Autonomous Driving
  8. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles