mlsanityData to 7 Oct 2026

Towards Knowledge-driven Autonomous Driving

arXiv 2312.04316 ↗First posted 7 Dec 2023cs.RO cs.AI cs.CVcited by 29

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

This paper explores the emerging knowledge-driven autonomous driving technologies. Our investigation highlights the limitations of current autonomous driving systems, in particular their sensitivity to data bias, difficulty in handling long-tail scenarios, and lack of interpretability. Conversely, knowledge-driven methods with the abilities of cognition, generalization and life-long learning emerge as a promising way to overcome these challenges. This paper delves into the essence of knowledge-driven autonomous driving and examines its core components: dataset & benchmark, environment, and driver agent. By leveraging large language models, world models, neural rendering, and other advanced artificial intelligence techniques, these components collectively contribute to a more holistic, adaptive, and intelligent autonomous driving system. The paper systematically organizes and reviews previous research efforts in this area, and provides insights and guidance for future research and practical applications of autonomous driving. We will continually share the latest updates on cutting-edge developments in knowledge-driven autonomous driving along with the relevant valuable open-source resources at: urlhttps://github.com/PJLab-ADG/awesome-knowledge-driven-AD.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 182 of its 276 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

182 of 276 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov

Other references · 181

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  13. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  14. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  15. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  16. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  17. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  18. 2023A Survey of Large Language Modelscited by 2,537
  19. 2015VQA: Visual Question Answeringcited by 2,489
  20. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  21. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  22. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  23. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  24. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  25. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  26. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  27. 2018World Modelscited by 1,444
  28. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  29. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  30. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  31. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  32. 2022Planning-oriented Autonomous Drivingcited by 1,068
  33. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  34. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  35. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  36. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  37. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  38. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711

50 most cited of 181 All 181 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 12; 2026: 1.

29 papers.

Sort:

By area:

  1. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  2. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  3. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  4. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  5. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Modelcited by 28
  6. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  7. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  8. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  9. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 19
  10. 2024PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Drivingcited by 17
  11. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Modelscited by 16
  12. 2025V2X-UniPool: Unifying Multimodal Perception and Knowledge Reasoning for Autonomous Drivingcited by 14
17 more
  1. 2025LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinkingcited by 13
  2. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  3. 2024Driving with Regulation: Trustworthy and Interpretable Decision-Making for Autonomous Driving with Retrieval-Augmented Reasoningcited by 11
  4. 2025OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Criticcited by 11
  5. 2024SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Modelscited by 10
  6. 2025SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Drivingcited by 9
  7. 2025DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Modelscited by 7
  8. 2024Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoningcited by 6
  9. 2024Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Surveycited by 6
  10. 2024Levels of AI Agents: from Rules to Large Language Modelscited by 6
  11. 2024CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learningcited by 5
  12. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  13. 2024Physics Enhanced Residual Policy Learning (PERPL) for safety cruising in mixed traffic platooning under actuator and communication delaycited by 1
  14. 2025Knowledge Integration Strategies in Autonomous Vehicle Prediction and Planning: A Comprehensive Surveycited by 1
  15. 2025A Knowledge-Informed Deep Learning Paradigm for Generalizable and Stability-Optimized Car-Following Modelscited by 1
  16. 2025LimSim Series: An Autonomous Driving Simulation Platform for Validation and Enhancementcited by 0
  17. 2026Agent-driven Long-tail Simulation for Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Large Language Model-empowered Autonomous Driving
  2. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions
  3. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
  4. 2026A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms
  5. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
  6. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  7. 2025A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routing
  8. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform