mlsanityData to 7 Oct 2026

RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model

arXiv 2402.10828 ↗First posted 16 Feb 2024cs.RO cs.AIcited by 91

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 2 more
Abstract (arXiv)

We need to trust robots that use often opaque AI methods. They need to explain themselves to us, and we need to trust their explanation. In this regard, explainability plays a critical role in trustworthy autonomous decision-making to foster transparency and acceptance among end users, especially in complex autonomous driving. Recent advancements in Multi-Modal Large Language models (MLLMs) have shown promising potential in enhancing the explainability as a driving agent by producing control predictions along with natural language explanations. However, severe data scarcity due to expensive annotation costs and significant domain gaps between different datasets makes the development of a robust and generalisable system an extremely challenging task. Moreover, the prohibitively expensive training requirements of MLLM and the unsolved problem of catastrophic forgetting further limit their generalisability post-deployment. To address these challenges, we present RAG-Driver, a novel retrieval-augmented multi-modal large language model that leverages in-context learning for high-performance, explainable, and generalisable autonomous driving. By grounding in retrieved expert demonstration, we empirically validate that RAG-Driver achieves state-of-the-art performance in producing driving action explanations, justifications, and control signal prediction. More importantly, it exhibits exceptional zero-shot generalisation capabilities to unseen environments without further training endeavours.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 79 of its 94 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

79 of 94 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023LingoQA: Visual Question Answering for Autonomous Driving21 Dec
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  3. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec
  4. 2023Dolphins: Multimodal Language Model for Driving1 Dec
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 74

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
38 more
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2022Planning-oriented Autonomous Drivingcited by 1,068
  10. 2020In Search of Lost Domain Generalizationcited by 1,042
  11. 2016End to End Learning for Self-Driving Carscited by 1,000
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  13. 2023Large Language Modelscited by 774
  14. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  15. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  16. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  17. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  18. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  19. 2014Deep metric learning using Triplet networkcited by 421
  20. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  21. 2019Learning by Cheatingcited by 361
  22. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  23. 2018Learning to Drive in a Daycited by 266
  24. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  25. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  26. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  27. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  28. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  29. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  30. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  31. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  32. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  33. 2023A Language Agent for Autonomous Drivingcited by 161
  34. 2015Sequence to Sequence -- Video to Textcited by 157
  35. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  36. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  37. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  38. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 47; 2026: 22.

91 papers.

Sort:

By area:

  1. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  3. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  4. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  5. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  6. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  7. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  8. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  9. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  10. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Drivingcited by 34
  11. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  12. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
38 more
  1. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Modelcited by 28
  2. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  3. 2025BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Drivingcited by 26
  4. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  5. 2025Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learningcited by 25
  6. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  7. 2024VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenescited by 22
  8. 2025SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Modelscited by 22
  9. 2024kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabulariescited by 20
  10. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
  11. 2025RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Drivingcited by 18
  12. 2025Can Multimodal Large Language Models be Guided to Improve Industrial Anomaly Detection?cited by 17
  13. 2025DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planningcited by 17
  14. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 17
  15. 2024Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Modelcited by 14
  16. 2025V2X-UniPool: Unifying Multimodal Perception and Knowledge Reasoning for Autonomous Drivingcited by 14
  17. 2025LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinkingcited by 13
  18. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  19. 2025FASIONAD++ : Integrating High-Level Instruction and Information Bottleneck in FAt-Slow fusION Systems for Enhanced Safety in Autonomous Driving with Adaptive Feedbackcited by 12
  20. 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agentscited by 12
  21. 2025OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Criticcited by 11
  22. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarioscited by 11
  23. 2024DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Drivingcited by 10
  24. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  25. 2025WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Drivingcited by 10
  26. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 10
  27. 2025SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Drivingcited by 9
  28. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  29. 2024MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarioscited by 8
  30. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  31. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 7
  32. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  33. 2025LawPal : A Retrieval Augmented Generation Based System for Enhanced Legal Accessibility in Indiacited by 6
  34. 2025BEV-LLM: Leveraging Multimodal BEV Maps for Scene Captioning in Autonomous Drivingcited by 6
  35. 2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
  36. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  37. 2025A Comprehensive LLM-powered Framework for Driving Intelligence Evaluationcited by 5
  38. 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Drivingcited by 5

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving
  2. 2023Personalized Autonomous Driving with Large Language Models: Field Experiments
  3. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models
  4. 2024RAG-based Explainable Prediction of Road Users Behaviors for Automated Driving using Knowledge Graphs and Large Language Models
  5. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  6. 2026Rethinking Language's Role in Efficient VLA for Autonomous Vehicles: Toward Smarter, Trustworthy Driving
  7. 2024RAG-Modulo: Solving Sequential Tasks using Experience, Critics, and Language Models
  8. 2025AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving