mlsanityData to 7 Oct 2026

Asynchronous Large Language Model Enhanced Planner for Autonomous Driving

arXiv 2406.14556 ↗First posted 20 Jun 2024cs.RO cs.CVcited by 38

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Despite real-time planners exhibiting remarkable performance in autonomous driving, the growing exploration of Large Language Models (LLMs) has opened avenues for enhancing the interpretability and controllability of motion planning. Nevertheless, LLM-based planners continue to encounter significant challenges, including elevated resource consumption and extended inference times, which pose substantial obstacles to practical deployment. In light of these challenges, we introduce AsyncDriver, a new asynchronous LLM-enhanced closed-loop framework designed to leverage scene-associated instruction features produced by LLM to guide real-time planners in making precise and controllable trajectory predictions. On one hand, our method highlights the prowess of LLMs in comprehending and reasoning with vectorized scene data and a series of routing instructions, demonstrating its effective assistance to real-time planners. On the other hand, the proposed framework decouples the inference processes of the LLM and real-time planners. By capitalizing on the asynchronous nature of their inference frequencies, our approach have successfully reduced the computational cost introduced by LLM, while maintaining comparable performance. Experiments show that our approach achieves superior closed-loop evaluation performance on nuPlan's challenging scenarios.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 38 of its 49 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

38 of 49 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model16 Feb
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  3. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Models12 Dec
  4. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs7 Dec
  5. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspective28 Nov
  6. 2023A Survey on Multimodal Large Language Models for Autonomous Driving21 Nov
  7. 2023A Language Agent for Autonomous Driving17 Nov
  8. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving2 Nov
  9. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles12 Oct
  10. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  11. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  12. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
2 more
  1. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  2. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Planning-oriented Autonomous Drivingcited by 1,068
  5. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  6. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  7. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  8. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  9. 2018Learning to Drive in a Daycited by 266
  10. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  11. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  12. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
12 more
  1. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  2. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  3. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  4. 2018Baidu Apollo EM Motion Plannercited by 99
  5. 2023Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveyscited by 75
  6. 2023LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoningcited by 59
  7. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 50
  8. 2023From Prediction to Planning With Goal Conditioned Lane Graph Traversalscited by 45
  9. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 41
  10. 2023MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Unsignalized Intersectionscited by 38
  11. 2023Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous drivingcited by 30
  12. 2023Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skillscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 22; 2026: 14.

38 papers.

Sort:

By area:

  1. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  2. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  3. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  4. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  5. 2025SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Drivingcited by 21
  6. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  7. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  8. 2025AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Drivingcited by 10
  9. 2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
  10. 2025Planning Safety Trajectories with Dual-Phase, Physics-Informed, and Transportation Knowledge-Driven Large Language Modelscited by 5
  11. 2026HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Drivingcited by 5
  12. 2025SafePath: Conformal Prediction for Safe LLM-Based Autonomous Navigationcited by 4
26 more
  1. 2025CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Drivingcited by 4
  2. 2025NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Drivingcited by 3
  3. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  4. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  5. 2026RAD-LAD: Rule and Language Grounded Autonomous Driving in Real-Timecited by 3
  6. 2025Urban Computing in the Era of Large Language Modelscited by 2
  7. 2025Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detectioncited by 2
  8. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Modelscited by 2
  9. 2026OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Modelscited by 2
  10. 2025A Knowledge-Informed Deep Learning Paradigm for Generalizable and Stability-Optimized Car-Following Modelscited by 1
  11. 2026Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planningcited by 1
  12. 2026ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Drivingcited by 1
  13. 2026Teaching Vision-Language-Action Models What to See and Where to Lookcited by 1
  14. 2025ManeuverGPT Agentic Control for Safe Autonomous Stunt Maneuverscited by 0
  15. 2025HyPlan: Hybrid Learning-Assisted Planning Under Uncertainty for Safe Autonomous Drivingcited by 0
  16. 2025UniPlanner: A Unified Motion Planning Framework for Autonomous Vehicle Decision-Making Systems via Multi-Dataset Integrationcited by 0
  17. 2025Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLMcited by 0
  18. 2025Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learningcited by 0
  19. 2026Towards Unconstrained Human-Object Interactioncited by 0
  20. 2026KG-ASG: Collision-Knowledge-Guided Closed-Loop Adversarial Scenario Generation With Primary-Support Attributioncited by 0
  21. 2026DrivingAgent: Design and Scheduling Agents for Autonomous Driving Systemscited by 0
  22. 2026Agent-driven Long-tail Simulation for Autonomous Drivingcited by 0
  23. 2026Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Drivingcited by 0
  24. 2026CommandLM: Data driven behavior level descriptor for ego vehiclescited by 0
  25. 2026PersonaDrive: Controllable Trajectory Prediction with Multi-Dimensional Driving Personascited by 0
  26. 2026FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoning
  2. 2025LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarios
  3. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignment
  4. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
  5. 2025Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving
  6. 2025Distilling Multi-modal Large Language Models for Autonomous Driving
  7. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  8. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs