mlsanityData to 7 Oct 2026

SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Driving

arXiv 2501.03535 ↗First posted 7 Jan 2025cs.AI cs.ROcited by 9

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 1 more
Abstract (arXiv)

This study addresses the critical need for enhanced situational awareness in autonomous driving (AD) by leveraging the contextual reasoning capabilities of large language models (LLMs). Unlike traditional perception systems that rely on rigid, label-based annotations, it integrates real-time, multimodal sensor data into a unified, LLMs-readable knowledge base, enabling LLMs to dynamically understand and respond to complex driving environments. To overcome the inherent latency and modality limitations of LLMs, a proactive Retrieval-Augmented Generation (RAG) is designed for AD, combined with a chain-of-thought prompting mechanism, ensuring rapid and context-rich understanding. Experimental results using real-world Vehicle-to-everything (V2X) datasets demonstrate significant improvements in perception and prediction performance, highlighting the potential of this framework to enhance safety, adaptability, and decision-making in next-generation AD systems.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 20 of its 33 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

20 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Driving2 Dec

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2019Object Detection in 20 Years: A Surveycited by 528
  4. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  5. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  6. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  7. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  8. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  9. 2023Collaborative Perception in Autonomous Driving: Methods, Datasets and Challengescited by 87
  10. 2024Communication-Efficient Collaborative Perception via Information Filling with Codebookcited by 67
  11. 2023Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perceptioncited by 38
  12. 2024Towards Optimizing the Costs of LLM Usagecited by 33
7 more
  1. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  2. 2024SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Modelscited by 28
  3. 2022Collaborative Perception for Autonomous Driving: Current Status and Future Trendcited by 10
  4. 2024Cross-Domain Transfer Learning using Attention Latent Features for Multi-Agent Trajectory Predictioncited by 5
  5. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 3
  6. 2021Set-Membership Estimation in Shared Situational Awareness for Automated Vehicles in Occluded Scenarioscited by 2
  7. 2024Energy-efficient Hybrid Model Predictive Trajectory Planning for Autonomous Electric Vehiclescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 1.

9 papers.

  1. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  2. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  3. 2025LangCoop: Collaborative Driving with Languagecited by 25
  4. 2025AirV2X: Unified Air-Ground Vehicle-to-Everything Collaborationcited by 11
  5. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  6. 2025Automated Vehicles Should be Connected with Natural Languagecited by 5
  7. 2025Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidancecited by 3
  8. 2025KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Modelscited by 1
  9. 2026Virtual Traffic Police: Large Language Model-Augmented Traffic Signal Control for Unforeseen Incidentscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Leveraging Large Language Models for Enhancing Autonomous Vehicle Perception
  2. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
  3. 2025V2X-UniPool: Unifying Multimodal Perception and Knowledge Reasoning for Autonomous Driving
  4. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
  5. 2025Logic-RAG: Augmenting Large Multimodal Models with Visual-Spatial Knowledge for Road Scene Understanding
  6. 2025Case-based Reasoning Augmented Large Language Model Framework for Decision Making in Realistic Safety-Critical Driving Scenarios
  7. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles
  8. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving