# RAG-based Explainable Prediction of Road Users Behaviors for Automated Driving using Knowledge Graphs and Large Language Models

arXiv: https://arxiv.org/abs/2405.00449 · HTML: https://mlsanity.com/paper/2405.00449/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 1 May 2024.

## Placed in

- End-to-end autonomous driving / [Large language models for driving](https://mlsanity.com/driving/large-language-models-for-driving/)

## Builds on (most cited first, 18 of 18)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2303.13948/) (2023)
- [Pedestrian Action Anticipation using Contextual Feature Fusion in Stacked RNNs](https://mlsanity.com/paper/2005.06582/) (2020)
- [Learning Spatiotemporal Features with 3D Convolutional Networks](https://mlsanity.com/paper/1412.0767/) (2014)
- [YOLOPv2: Better, Faster, Stronger for Panoptic Driving Perception](https://mlsanity.com/paper/2208.11434/) (2022)
- [A Survey of Knowledge Graph Embedding and Their Applications](https://mlsanity.com/paper/2107.07842/) (2021)
- [PSI: A Benchmark for Human Interpretation and Response in Traffic Interactions](https://mlsanity.com/paper/2112.02604/) (2021)
- [An Evaluation of Knowledge Graph Embeddings for Autonomous Driving Data: Experience and Practice](https://mlsanity.com/paper/2003.00344/) (2020)
- [Vehicle Lane Change Prediction based on Knowledge Graph Embeddings and Bayesian Inference](https://mlsanity.com/paper/2312.06336/) (2023)
- [Vehicle Trajectory Prediction on Highways Using Bird Eye View Representations and Deep Learning](https://mlsanity.com/paper/2207.01407/) (2022)
- [PedRecNet: Multi-task deep neural network for full 3D human pose and orientation estimation](https://mlsanity.com/paper/2204.11548/) (2022)
- [Analysis over vision-based models for pedestrian action anticipation](https://mlsanity.com/paper/2305.17451/) (2023)
- [Experimental Insights Towards Explainable and Interpretable Pedestrian Crossing Prediction](https://mlsanity.com/paper/2312.02872/) (2023)

## Built on by (most cited first, 20 of 20)

- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Driving](https://mlsanity.com/paper/2503.13861/) (2025)
- [Large Language Models for Mobility Analysis in Transportation Systems: A Survey on Forecasting Tasks](https://mlsanity.com/paper/2405.02357/) (2024)
- [Knowledge Graph-extended Retrieval Augmented Generation for Question Answering](https://mlsanity.com/paper/2504.08893/) (2025)
- [Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education](https://mlsanity.com/paper/2412.14191/) (2024)
- [RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Models](https://mlsanity.com/paper/2412.11050/) (2024)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [Spatial Retrieval Augmented Autonomous Driving](https://mlsanity.com/paper/2512.06865/) (2025)
- [Driving-RAG: Driving Scenarios Embedding, Search, and RAG Applications](https://mlsanity.com/paper/2504.04419/) (2025)
- [The Era of End-to-End Autonomy: Transitioning from Rule-Based Driving to Large Driving Models](https://mlsanity.com/paper/2603.16050/) (2026)
- [Explainable Lane Change Prediction for Near-Crash Scenarios Using Knowledge Graph Embeddings and Retrieval Augmented Generation](https://mlsanity.com/paper/2501.11560/) (2025)
- [Divide by Question, Conquer by Agent: SPLIT-RAG with Question-Driven Graph Partitioning](https://mlsanity.com/paper/2505.13994/) (2025)
- [Design Insights and Comparative Evaluation of a Hardware-Based Cooperative Perception Architecture for Lane Change Prediction](https://mlsanity.com/paper/2509.20218/) (2025)
- [VLADriver-RAG: Retrieval-Augmented Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2605.08133/) (2026)
- [Visual Reasoning at Urban Intersections: FineTuning GPT-4o for Traffic Conflict Detection](https://mlsanity.com/paper/2502.20573/) (2025)
- [Human-like Semantic Navigation for Autonomous Driving using Knowledge Representation and Large Language Models](https://mlsanity.com/paper/2505.16498/) (2025)
- [Information Coordination as a Bridge: A Neuro-Symbolic Architecture for Reliable Autonomous Driving Scene Understanding](https://mlsanity.com/paper/2605.04475/) (2026)
- [TCAR-Gen: Temporal Graph Retrieval with Evidence Fusion for Knowledge-Grounded Generation](https://mlsanity.com/paper/2606.00029/) (2026)
- [Beyond Retrieval Relevance: Scene-Grounded Risk Entailment for Vision-Language Driving](https://mlsanity.com/paper/2609.34145/) (2026)

## Abstract (arXiv)

Prediction of road users' behaviors in the context of autonomous driving has gained considerable attention by the scientific community in the last years. Most works focus on predicting behaviors based on kinematic information alone, a simplification of the reality since road users are humans, and as such they are highly influenced by their surrounding context. In addition, a large plethora of research works rely on powerful Deep Learning techniques, which exhibit high performance metrics in prediction tasks but may lack the ability to fully understand and exploit the contextual semantic information contained in the road scene, not to mention their inability to provide explainable predictions that can be understood by humans. In this work, we propose an explainable road users' behavior prediction system that integrates the reasoning abilities of Knowledge Graphs (KG) and the expressiveness capabilities of Large Language Models (LLM) by using Retrieval Augmented Generation (RAG) techniques. For that purpose, Knowledge Graph Embeddings (KGE) and Bayesian inference are combined to allow the deployment of a fully inductive reasoning system that enables the issuing of predictions that rely on legacy information contained in the graph as well as on current evidence gathered in real time by onboard sensors. Two use cases have been implemented following the proposed approach: 1) Prediction of pedestrians' crossing actions; 2) Prediction of lane change maneuvers. In both cases, the performance attained surpasses the current state of the art in terms of anticipation and F1-score, showing a promising avenue for future research in this field.