mlsanityData to 7 Oct 2026

Making Large Language Models Better Planners with Reasoning-Decision Alignment

arXiv 2408.13890 ↗First posted 25 Aug 2024cs.CVcited by 51

On the map: LLM reasoning → Chain-of-thought reasoning for driving

Abstract (arXiv)

Data-driven approaches for autonomous driving (AD) have been widely adopted in the past decade but are confronted with dataset bias and uninterpretability. Inspired by the knowledge-driven nature of human driving, recent approaches explore the potential of large language models (LLMs) to improve understanding and decision-making in traffic scenarios. They find that the pretrain-finetune paradigm of LLMs on downstream data with the Chain-of-Thought (CoT) reasoning process can enhance explainability and scene understanding. However, such a popular strategy proves to suffer from the notorious problems of misalignment between the crafted CoTs against the consequent decision-making, which remains untouched by previous LLM-based AD methods. To address this problem, we motivate an end-to-end decision-making model based on multimodality-augmented LLM, which simultaneously executes CoT reasoning and carries out planning results. Furthermore, we propose a reasoning-decision alignment constraint between the paired CoTs and planning results, imposing the correspondence between reasoning and decision-making. Moreover, we redesign the CoTs to enable the model to comprehend complex scenarios and enhance decision-making performance. We dub our proposed large language planners with reasoning-decision alignment as RDA-Driver. Experimental evaluations on the nuScenes and DriveLM-nuScenes benchmarks demonstrate the effectiveness of our RDA-Driver in enhancing the performance of end-to-end AD systems. Specifically, our RDA-Driver achieves state-of-the-art planning performance on the nuScenes dataset with 0.80 L2 error and 0.32 collision rate, and also achieves leading results on challenging DriveLM-nuScenes benchmarks with 0.82 L2 error and 0.38 collision rate.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 48 of its 53 references that are in the map. First posted on arXiv 25 Aug 2024.

Builds on

48 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  11. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
35 more
  1. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  2. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  3. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  4. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  5. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  6. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  7. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  8. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  11. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  12. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  13. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  14. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  15. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  16. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  17. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  18. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  19. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  20. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  21. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  22. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  23. 2023A Language Agent for Autonomous Drivingcited by 161
  24. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  25. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  26. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  27. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  28. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  29. 2022Differentiable Raycasting for Self-supervised Occupancy Forecastingcited by 83
  30. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  31. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63
  32. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 41
  33. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  34. 2022Path-Aware Graph Attention for HD Maps in Motion Predictioncited by 23
  35. 2023BEVGPT: Generative Pre-trained Large Model for Autonomous Driving Prediction, Decision-Making, and Planningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 27; 2026: 22.

51 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  3. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  4. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  5. 2024VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulationcited by 45
  6. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  7. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  8. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  9. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  10. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Drivingcited by 26
  11. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  12. 2025TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planningcited by 21
38 more
  1. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  2. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 18
  3. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  4. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 17
  5. 2025InsightDrive: Insight Scene Representation for End-to-End Autonomous Drivingcited by 14
  6. 2025VDT-Auto: End-to-end Autonomous Driving with VLM-Guided Diffusion Transformerscited by 12
  7. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  8. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  9. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  10. 2026MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Drivingcited by 10
  11. 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Drivingcited by 9
  12. 2025A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Drivingcited by 5
  13. 2025VDRive: Leveraging Reinforced VLA and Diffusion Policy for End-to-end Autonomous Drivingcited by 5
  14. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planningcited by 5
  15. 2025A Unified Perception-Language-Action Framework for Adaptive Autonomous Drivingcited by 4
  16. 2025RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Casecited by 4
  17. 2026SparseOccVLA: Bridging Occupancy and Vision-Language Models via Sparse Queries for Unified 4D Scene Understanding and Planningcited by 4
  18. 2026Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planningcited by 4
  19. 2025MINDSTORES: Memory-Informed Neural Decision Synthesis for Task-Oriented Reinforcement in Embodied Systemscited by 3
  20. 2025PRIMEDrive-CoT: A Precognitive Chain-of-Thought Framework for Uncertainty-Aware Object Interaction in Driving Scene Scenariocited by 3
  21. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  22. 2025DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Riskscited by 3
  23. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  24. 2026VLGA: Vision-Language-Geometry-Action Models for Autonomous Drivingcited by 3
  25. 2025TrajMoE: Scene-Adaptive Trajectory Planning with Mixture of Experts and Reinforcement Learningcited by 1
  26. 2026Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planningcited by 1
  27. 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1
  28. 2026Pedestrian-Aware LLM-Driven Behavioral Planning for Autonomous Vehiclescited by 1
  29. 2026Grounding Driving VLA via Inverse Kinematicscited by 1
  30. 2025Colon-X: Advancing Intelligent Colonoscopy toward Clinical Reasoningcited by 0
  31. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representationscited by 0
  32. 2026Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous Vehiclescited by 0
  33. 2026TPS-Drive: Task-Guided Representation Purification for VLM-based Autonomous Drivingcited by 0
  34. 2026VeriDrive: Verifiable Counterfactual Supervision for Cost-Efficient Vision-Language Planningcited by 0
  35. 2026FactorDrive: Adaptive Multi-Step Reasoning Driven by Planning-Critical Factors for End-to-End Autonomous Drivingcited by 0
  36. 2026PersonaDrive: Controllable Trajectory Prediction with Multi-Dimensional Driving Personascited by 0
  37. 2026A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Drivingcited by 0
  38. 2026Sometimes You Gotta Run Before You Can Walk: Run-then-Walk Scheduling Strategy for VLM Autonomous Drivingcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  2. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  3. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving
  4. 2026Large Language Model based Interactive Decision-Making for Autonomous Driving
  5. 2026Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving
  6. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
  7. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  8. 2025LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Driving