mlsanityData to 7 Oct 2026

VLADriver-RAG: Retrieval-Augmented Vision-Language-Action Models for Autonomous Driving

arXiv 2605.08133 ↗First posted 1 May 2026cs.CV cs.AIcited by 0

On the map: Vision-language-action models → VLA for autonomous driving

also in 2 more
Abstract (arXiv)

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving, yet their reliance on implicit parametric knowledge limits generalization in long-tail scenarios. While Retrieval-Augmented Generation (RAG) offers a solution by accessing external expert priors, standard visual retrieval suffers from high latency and semantic ambiguity. To address these challenges, we propose VLADriver-RAG, a framework that grounds planning in explicit, structure-aware historical knowledge. Specifically, we abstract sensory inputs into spatiotemporal semantic graphs via a textitVisual-to-Scenario mechanism, effectively filtering visual noise. To ensure retrieval relevance, we employ a textitScenario-Aligned Embedding Model that utilizes Graph-DTW metric alignment to prioritize intrinsic topological consistency over superficial visual similarity. These retrieved priors are then fused within a query-based VLA backbone to synthesize precise, disentangled trajectories. Extensive experiments on the Bench2Drive benchmark establish a new state-of-the-art, achieving a Driving Score of 89.12.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLA for autonomous driving, one of 13 directions in Vision-language-action models. Builds on 32 of its 36 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

32 of 36 references.

References · 32

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  7. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2022Planning-oriented Autonomous Drivingcited by 1,068
  10. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  11. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  12. 2023Active Retrieval Augmented Generationcited by 506
20 more
  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  3. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  4. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  5. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  6. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  7. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  8. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  9. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  10. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  11. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  12. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  13. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  14. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  15. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  16. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  17. 2024RAG-based Explainable Prediction of Road Users Behaviors for Automated Driving using Knowledge Graphs and Large Language Modelscited by 20
  18. 2024VistaScenario: Interaction Scenario Engineering for Vehicles with Intelligent Systems for Transport Automationcited by 5
  19. 2025Spatial Retrieval Augmented Autonomous Drivingcited by 5
  20. 2025Driving-RAG: Driving Scenarios Embedding, Search, and RAG Applicationscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  2. 2025VLA-R: Vision-Language Action Retrieval toward Open-World End-to-End Autonomous Driving
  3. 2026DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving
  4. 2025RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Driving
  5. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving
  6. 2026Unifying Language-Action Understanding and Generation for Autonomous Driving
  7. 2026S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving
  8. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving