mlsanityData to 7 Oct 2026

LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning

arXiv 2401.00125 ↗First posted 30 Dec 2023cs.AI cs.CVcited by 59

On the map: LLM reasoning → Chain-of-thought reasoning for driving

Abstract (arXiv)

Although planning is a crucial component of the autonomous driving stack, researchers have yet to develop robust planning algorithms that are capable of safely handling the diverse range of possible driving scenarios. Learning-based planners suffer from overfitting and poor long-tail performance. On the other hand, rule-based planners generalize well, but might fail to handle scenarios that require complex driving maneuvers. To address these limitations, we investigate the possibility of leveraging the common-sense reasoning capabilities of Large Language Models (LLMs) such as GPT4 and Llama2 to generate plans for self-driving vehicles. In particular, we develop a novel hybrid planner that leverages a conventional rule-based planner in conjunction with an LLM-based planner. Guided by commonsense reasoning abilities of LLMs, our approach navigates complex scenarios which existing planners struggle with, produces well-reasoned outputs while also remaining grounded through working alongside the rule-based approach. Through extensive evaluation on the nuPlan benchmark, we achieve state-of-the-art performance, outperforming all existing pure learning- and rule-based methods across most metrics. Our code will be available at https://llmassist.github.io.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 29 of its 37 references that are in the map. First posted on arXiv 30 Dec 2023.

Builds on

29 of 37 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2022Planning-oriented Autonomous Drivingcited by 1,068
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
17 more
  1. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  2. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  3. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  4. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  5. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  6. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  7. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  8. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  9. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  10. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  11. 2018Baidu Apollo EM Motion Plannercited by 99
  12. 2021LookOut: Diverse Multi-Future Prediction and Planning for Self-Drivingcited by 96
  13. 2023Improving Language Models via Plug-and-Play Retrieval Feedbackcited by 87
  14. 2020Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Drivingcited by 24
  15. 2023Tree-structured Policy Planning with Learned Behavior Modelscited by 24
  16. 2021Contingencies from Observations: Tractable Contingency Planning with Learned Behavior Modelscited by 22
  17. 2023MBAPPE: MCTS-Built-Around Prediction for Planning Explicitlycited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 23; 2026: 6.

59 papers.

Sort:

By area:

  1. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  2. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  3. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  4. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  5. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  6. 2024Agents Thinking Fast and Slow: A Talker-Reasoner Architecturecited by 35
  7. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  8. 2024LLM-Assisted Light: Leveraging Large Language Model Capabilities for Human-Mimetic Traffic Signal Control in Complex Urban Environmentscited by 31
  9. 2025Safety Guardrails for LLM-Enabled Robotscited by 30
  10. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  11. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  12. 2024VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenescited by 22
38 more
  1. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  2. 2024LLM-based Conversational AI Therapist for Daily Functioning Screening and Psychotherapeutic Intervention via Everyday Smart Devicescited by 21
  3. 2024LightVA: Lightweight Visual Analytics with LLM Agent-Based Task Planning and Executioncited by 20
  4. 2024SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environmentscited by 16
  5. 2024A Survey on Human-Centric LLMscited by 13
  6. 2024Instruct Large Language Models to Drive like Humanscited by 11
  7. 2024Planning with Adaptive World Models for Autonomous Drivingcited by 11
  8. 2025Air-Ground Collaboration for Language-Specified Missions in Unknown Environmentscited by 11
  9. 2024Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Modelscited by 10
  10. 2025Distilling On-device Language Models for Robot Planning with Minimal Human Interventioncited by 10
  11. 2025Safety of Embodied Navigation: A Surveycited by 9
  12. 2025A Roadmap to Guide the Integration of LLMs in Hierarchical Planningcited by 7
  13. 2025V3LMA: Visual 3D-enhanced Language Model for Autonomous Drivingcited by 7
  14. 2025LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarioscited by 7
  15. 2024Enhancing LLM-based Autonomous Driving Agents to Mitigate Perception Attackscited by 6
  16. 2025Toward Super Agent System with Hybrid AI Routerscited by 6
  17. 2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
  18. 2024CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long-Range Reasoning Problems using LLMscited by 5
  19. 2024Dual-AEB: Synergizing Rule-Based and Multimodal Large Language Models for Effective Emergency Brakingcited by 5
  20. 2025A Comprehensive LLM-powered Framework for Driving Intelligence Evaluationcited by 5
  21. 2025Planning Safety Trajectories with Dual-Phase, Physics-Informed, and Transportation Knowledge-Driven Large Language Modelscited by 5
  22. 2025LLMAP: LLM-Assisted Multi-Objective Route Planning with User Preferencescited by 5
  23. 2024LoRD: Adapting Differentiable Driving Policies to Distribution Shiftscited by 4
  24. 2024Jailbreaking LLM-Controlled Robotscited by 4
  25. 2024GRS-QA -- Graph Reasoning-Structured Question Answering Datasetcited by 4
  26. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 3
  27. 2024Autoware.Flex: Human-Instructed Dynamically Reconfigurable Autonomous Driving Systemscited by 3
  28. 2025Driving-RAG: Driving Scenarios Embedding, Search, and RAG Applicationscited by 3
  29. 2026RAD-LAD: Rule and Language Grounded Autonomous Driving in Real-Timecited by 3
  30. 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
  31. 2024Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networkscited by 1
  32. 2024Think-on-Process: Dynamic Process Generation for Collaborative Development of Multi-Agent Systemcited by 1
  33. 2025Knowledge Integration Strategies in Autonomous Vehicle Prediction and Planning: A Comprehensive Surveycited by 1
  34. 2025Unlock the Power of Unlabeled Data in Language Driving Modelcited by 1
  35. 2025RoboPARA: Dual-Arm Robot Planning with Parallel Allocation and Recomposition Across Taskscited by 1
  36. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observationscited by 1
  37. 2026ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Drivingcited by 1
  38. 2024SciQu: Accelerating Materials Properties Prediction with Automated Literature Mining for Self-Driving Laboratoriescited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  3. 2025LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Driving
  4. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignment
  5. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving
  6. 2023GPT-Driver: Learning to Drive with GPT
  7. 2025AuDeRe: Automated Strategy Decision and Realization in Robot Planning and Control via LLMs
  8. 2025LLMs Can Plan Only If We Tell Them