# Driving-RAG: Driving Scenarios Embedding, Search, and RAG Applications

arXiv: https://arxiv.org/abs/2504.04419 · HTML: https://mlsanity.com/paper/2504.04419/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 6 Apr 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)
- End-to-end autonomous driving / [Large language models for driving](https://mlsanity.com/driving/large-language-models-for-driving/)

## Builds on (most cited first, 18 of 18)

- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model](https://mlsanity.com/paper/2402.10828/) (2024)
- [LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning](https://mlsanity.com/paper/2401.00125/) (2023)
- [CitySim: A Drone-Based Vehicle Trajectory Dataset for Safety Oriented Research and Digital Twins](https://mlsanity.com/paper/2208.11036/) (2022)
- [RAG-based Explainable Prediction of Road Users Behaviors for Automated Driving using Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2405.00449/) (2024)
- [Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models](https://mlsanity.com/paper/2309.04316/) (2023)
- [Scenario-Transferable Semantic Graph Reasoning for Interaction-Aware Probabilistic Prediction](https://mlsanity.com/paper/2004.03053/) (2020)
- [RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding](https://mlsanity.com/paper/2304.08600/) (2023)
- [roadscene2vec: A Tool for Extracting and Embedding Road Scene-Graphs](https://mlsanity.com/paper/2109.01183/) (2021)
- [Online Product Quantization](https://mlsanity.com/paper/1711.10775/) (2017)
- [Clustering of Driving Encounter Scenarios Using Connected Vehicle Trajectories](https://mlsanity.com/paper/1807.08415/) (2018)
- [Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models](https://mlsanity.com/paper/2406.04300/) (2024)
- [Traffic Scene Similarity: a Graph-based Contrastive Learning Approach](https://mlsanity.com/paper/2309.09720/) (2023)
- [VistaScenario: Interaction Scenario Engineering for Vehicles with Intelligent Systems for Transport Automation](https://mlsanity.com/paper/2402.07720/) (2024)

## Built on by (most cited first, 3 of 3)

- [RealDrive: Retrieval-Augmented Driving with Diffusion Models](https://mlsanity.com/paper/2505.24808/) (2025)
- [Virtual Traffic Police: Large Language Model-Augmented Traffic Signal Control for Unforeseen Incidents](https://mlsanity.com/paper/2601.15816/) (2026)
- [VLADriver-RAG: Retrieval-Augmented Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2605.08133/) (2026)

## Abstract (arXiv)

Driving scenario data play an increasingly vital role in the development of intelligent vehicles and autonomous driving. Accurate and efficient scenario data search is critical for both online vehicle decision-making and planning, and offline scenario generation and simulations, as it allows for leveraging the scenario experiences to improve the overall performance. Especially with the application of large language models (LLMs) and Retrieval-Augmented-Generation (RAG) systems in autonomous driving, urgent requirements are put forward. In this paper, we introduce the Driving-RAG framework to address the challenges of efficient scenario data embedding, search, and applications for RAG systems. Our embedding model aligns fundamental scenario information and scenario distance metrics in the vector space. The typical scenario sampling method combined with hierarchical navigable small world can perform efficient scenario vector search to achieve high efficiency without sacrificing accuracy. In addition, the reorganization mechanism by graph knowledge enhances the relevance to the prompt scenarios and augment LLM generation. We demonstrate the effectiveness of the proposed framework on typical trajectory planning task for complex interactive scenarios such as ramps and intersections, showcasing its advantages for RAG applications.