mlsanityData to 7 Oct 2026

A Language Agent for Autonomous Driving

arXiv 2311.10813 ↗First posted 17 Nov 2023cs.CV cs.AI cs.CLcited by 161

On the map: End-to-end autonomous driving → Large language models for driving

also in 1 more
Abstract (arXiv)

Human-level driving is an ultimate goal of autonomous driving. Conventional approaches formulate autonomous driving as a perception-prediction-planning framework, yet their systems do not capitalize on the inherent reasoning ability and experiential knowledge of humans. In this paper, we propose a fundamental paradigm shift from current pipelines, exploiting Large Language Models (LLMs) as a cognitive agent to integrate human-like intelligence into autonomous driving systems. Our approach, termed Agent-Driver, transforms the traditional autonomous driving pipeline by introducing a versatile tool library accessible via function calls, a cognitive memory of common sense and experiential knowledge for decision-making, and a reasoning engine capable of chain-of-thought reasoning, task planning, motion planning, and self-reflection. Powered by LLMs, our Agent-Driver is endowed with intuitive common sense and robust reasoning capabilities, thus enabling a more nuanced, human-like approach to autonomous driving. We evaluate our approach on the large-scale nuScenes benchmark, and extensive experiments substantiate that our Agent-Driver significantly outperforms the state-of-the-art driving methods by a large margin. Our approach also demonstrates superior interpretability and few-shot learning ability to these methods.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 37 of its 47 references that are in the map. First posted on arXiv 17 Nov 2023.

Builds on

37 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  3. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  4. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2022Planning-oriented Autonomous Drivingcited by 1,068
  10. 2020Convolutional Occupancy Networkscited by 760
  11. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  12. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
21 more
  1. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  2. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  3. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  4. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  5. 2021Voxel Transformer for 3D Object Detectioncited by 247
  6. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  7. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  8. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  9. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  10. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  11. 2023Scene as Occupancycited by 208
  12. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  13. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  14. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  15. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  16. 2018Baidu Apollo EM Motion Plannercited by 99
  17. 2022Differentiable Raycasting for Self-supervised Occupancy Forecastingcited by 83
  18. 2021Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detectioncited by 79
  19. 2018Conditional Affordance Learning for Driving in Urban Environmentscited by 70
  20. 2023Implicit Occupancy Flow Fields for Perception and Prediction in Self-Drivingcited by 30
  21. 2021Trajectory Prediction with Linguistic Representationscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 52; 2025: 70; 2026: 33.

161 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  2. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  4. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  5. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  6. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  7. 2024Behavior Generation with Latent Actionscited by 159
  8. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  9. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 144
  10. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  11. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 127
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
38 more
  1. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 113
  2. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  3. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  4. 2025How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behaviorcited by 95
  5. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  6. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 89
  7. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  8. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  9. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  10. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  11. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  12. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  13. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  14. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Usecited by 46
  15. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  16. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  17. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  18. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  19. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  20. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  21. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  22. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 32
  23. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  24. 2024Language-Image Models with 3D Understandingcited by 31
  25. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  26. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  27. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  28. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  29. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  30. 2024GraphAD: Interaction Scene Graph for End-to-end Autonomous Drivingcited by 25
  31. 2024Why Solving Multi-agent Path Finding with Large Language Model has not Succeeded Yetcited by 22
  32. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  33. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  34. 2024AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Mergingcited by 22
  35. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  36. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Drivingcited by 22
  37. 2025SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Modelscited by 22
  38. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms
  2. 2026Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving
  3. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles
  4. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  5. 2023SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Data
  6. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
  7. 2025DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving
  8. 2025DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking