# Asynchronous Large Language Model Enhanced Planner for Autonomous Driving

arXiv: https://arxiv.org/abs/2406.14556 · HTML: https://mlsanity.com/paper/2406.14556/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 38 of its 49 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Placed in

- End-to-end autonomous driving / [Large language models for driving](https://mlsanity.com/driving/large-language-models-for-driving/)

## Builds on (most cited first, 30 of 38)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023)
- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving](https://mlsanity.com/paper/2310.01957/) (2023)
- [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023)
- [LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving](https://mlsanity.com/paper/2310.03026/) (2023)
- [A Language Agent for Autonomous Driving](https://mlsanity.com/paper/2311.10813/) (2023)
- [Drive Like a Human: Rethinking Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2307.07162/) (2023)
- [GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving](https://mlsanity.com/paper/2303.05760/) (2023)
- [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023)
- [Dolphins: Multimodal Language Model for Driving](https://mlsanity.com/paper/2312.00438/) (2023)
- [Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving](https://mlsanity.com/paper/2312.03661/) (2023)
- [Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients](https://mlsanity.com/paper/2109.13333/) (2021)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Baidu Apollo EM Motion Planner](https://mlsanity.com/paper/1807.08048/) (2018)
- [RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model](https://mlsanity.com/paper/2402.10828/) (2024)
- [Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys](https://mlsanity.com/paper/2303.17220/) (2023)
- [Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2310.08034/) (2023)
- [LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning](https://mlsanity.com/paper/2401.00125/) (2023)

## Built on by (most cited first, 30 of 38)

- [A Survey on Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2506.24044/) (2025)
- [DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation](https://mlsanity.com/paper/2411.11252/) (2024)
- [NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2503.12772/) (2025)
- [VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Driving](https://mlsanity.com/paper/2408.04821/) (2024)
- [SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving](https://mlsanity.com/paper/2505.16805/) (2025)
- [Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving](https://mlsanity.com/paper/2503.21505/) (2025)
- [Multi-modal Knowledge Distillation-based Human Trajectory Forecasting](https://mlsanity.com/paper/2503.22201/) (2025)
- [AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving](https://mlsanity.com/paper/2511.06253/) (2025)
- [VLMPlanner: Integrating Visual Language Models with Motion Planning](https://mlsanity.com/paper/2507.20342/) (2025)
- [Planning Safety Trajectories with Dual-Phase, Physics-Informed, and Transportation Knowledge-Driven Large Language Models](https://mlsanity.com/paper/2504.04562/) (2025)
- [HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2602.13329/) (2026)
- [SafePath: Conformal Prediction for Safe LLM-Based Autonomous Navigation](https://mlsanity.com/paper/2505.09427/) (2025)
- [CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Driving](https://mlsanity.com/paper/2505.21581/) (2025)
- [NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.14589/) (2025)
- [Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges](https://mlsanity.com/paper/2512.00021/) (2025)
- [Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation](https://mlsanity.com/paper/2512.18028/) (2025)
- [RAD-LAD: Rule and Language Grounded Autonomous Driving in Real-Time](https://mlsanity.com/paper/2603.28522/) (2026)
- [OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models](https://mlsanity.com/paper/2604.17915/) (2026)
- [Urban Computing in the Era of Large Language Models](https://mlsanity.com/paper/2504.02009/) (2025)
- [Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detection](https://mlsanity.com/paper/2507.17456/) (2025)
- [From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models](https://mlsanity.com/paper/2512.05277/) (2025)
- [A Knowledge-Informed Deep Learning Paradigm for Generalizable and Stability-Optimized Car-Following Models](https://mlsanity.com/paper/2504.14241/) (2025)
- [Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning](https://mlsanity.com/paper/2602.07680/) (2026)
- [ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Driving](https://mlsanity.com/paper/2606.25509/) (2026)
- [Teaching Vision-Language-Action Models What to See and Where to Look](https://mlsanity.com/paper/2607.01658/) (2026)
- [ManeuverGPT Agentic Control for Safe Autonomous Stunt Maneuvers](https://mlsanity.com/paper/2503.09035/) (2025)
- [HyPlan: Hybrid Learning-Assisted Planning Under Uncertainty for Safe Autonomous Driving](https://mlsanity.com/paper/2510.07210/) (2025)
- [UniPlanner: A Unified Motion Planning Framework for Autonomous Vehicle Decision-Making Systems via Multi-Dataset Integration](https://mlsanity.com/paper/2510.24166/) (2025)
- [Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLM](https://mlsanity.com/paper/2511.14499/) (2025)
- [Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learning](https://mlsanity.com/paper/2512.05172/) (2025)

## Abstract (arXiv)

Despite real-time planners exhibiting remarkable performance in autonomous driving, the growing exploration of Large Language Models (LLMs) has opened avenues for enhancing the interpretability and controllability of motion planning. Nevertheless, LLM-based planners continue to encounter significant challenges, including elevated resource consumption and extended inference times, which pose substantial obstacles to practical deployment. In light of these challenges, we introduce AsyncDriver, a new asynchronous LLM-enhanced closed-loop framework designed to leverage scene-associated instruction features produced by LLM to guide real-time planners in making precise and controllable trajectory predictions. On one hand, our method highlights the prowess of LLMs in comprehending and reasoning with vectorized scene data and a series of routing instructions, demonstrating its effective assistance to real-time planners. On the other hand, the proposed framework decouples the inference processes of the LLM and real-time planners. By capitalizing on the asynchronous nature of their inference frequencies, our approach have successfully reduced the computational cost introduced by LLM, while maintaining comparable performance. Experiments show that our approach achieves superior closed-loop evaluation performance on nuPlan's challenging scenarios.