More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
On the map: LLM reasoning → Chain-of-thought reasoning for driving
also in 2 more
Abstract (arXiv)
Vision-Language Model (VLM) driving agents promise explainable end-to-end autonomy by first producing natural-language reasoning and then predicting trajectory planning. However, whether planning is causally driven by this reasoning remains a critical but unverified assumption. To investigate this, we build DriveMind, a large-scale driving Visual Question Answering corpus with plan-aligned Chain-of-Thought (CoT), automatically generated from nuPlan. Our data generation process converts sensors and annotations into structured inputs and, crucially, separates priors from to-be-reasoned signals, enabling clean information ablations. Using DriveMind, we train representative VLM agents with Supervised Fine-Tuning and Group Relative Policy Optimization and evaluate them with nuPlan's metrics. Our results, unfortunately, indicate a consistent causal disconnect in reasoning-planning: removing ego/navigation priors causes large drops in planning scores, whereas removing CoT produces only minor changes. Attention analysis further shows that planning primarily focuses on priors rather than the CoT. Based on this evidence, we propose the Reasoning-Planning Decoupling Hypothesis, positing that the training-yielded reasoning is an ancillary byproduct rather than a causal mediator. To enable efficient diagnosis, we introduce a novel, training-free probe that measures an agent's reliance on priors by evaluating its planning robustness against minor input perturbations. In summary, we provide the community with a new dataset and a diagnostic tool to evaluate the causal fidelity of future models.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 34 of its 43 references that are in the map. First posted on arXiv 6 Oct 2025.
Builds on
34 of 43 references.
Closest ancestors · 3
Same direction, newest first
- 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning23 Jun
- 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning6 Apr
- 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving25 Mar
Other references · 31
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2023Visual Instruction Tuningcited by 7,792
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2022Planning-oriented Autonomous Drivingcited by 1,068
- 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
- 2024A Survey on Large Language Models for Code Generationcited by 646
19 more
- 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
- 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
- 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
- 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
- 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
- 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
- 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
- 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
- 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
- 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
- 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
- 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
- 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 103
- 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
- 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
- 2024End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotationcited by 27
- 2024Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Modelscited by 25
- 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
- 2025A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulatorcited by 7
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving
- 2026DriveMA: Rethinking Language Interfaces in Driving VLAs with One-Step Meta-Actions
- 2026Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
- 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation
- 2026AutoDrive-P3: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning
- 2026Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs
- 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
- 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving