mlsanityData to 14 Oct 2026

Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving

arXiv 2312.03661 ↗First posted 6 Dec 2023cs.CVcited by 123

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 2 more
Abstract (arXiv)

Large vision-language models (VLMs) have garnered increasing interest in autonomous driving areas, due to their advanced capabilities in complex reasoning tasks essential for highly autonomous vehicle behavior. Despite their potential, research in autonomous systems is hindered by the lack of datasets with annotated reasoning chains that explain the decision-making processes in driving. To bridge this gap, we present Reason2Drive, a benchmark dataset with over 600K video-text pairs, aimed at facilitating the study of interpretable reasoning in complex driving environments. We distinctly characterize the autonomous driving process as a sequential combination of perception, prediction, and reasoning steps, and the question-answer pairs are automatically collected from a diverse range of open-source outdoor driving datasets, including nuScenes, Waymo and ONCE. Moreover, we introduce a novel aggregated evaluation metric to assess chain-based reasoning performance in autonomous systems, addressing the semantic ambiguities of existing metrics such as BLEU and CIDEr. Based on the proposed benchmark, we conduct experiments to assess various existing VLMs, revealing insights into their reasoning capabilities. Additionally, we develop an efficient approach to empower VLMs to leverage object-level perceptual elements in both feature extraction and prediction, further enhancing their reasoning accuracy. The code and dataset will be released.

Key facts. Cited by 123 papers in the map (data through 14 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 43 of its 51 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

43 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2017Mask R-CNNcited by 7,277
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
30 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  7. 2022Planning-oriented Autonomous Drivingcited by 1,087
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  9. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  10. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  11. 2023Instruction Tuning with GPT-4cited by 686
  12. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  13. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  14. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  15. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  16. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  17. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 177
  18. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  19. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  20. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  21. 2023Language Prompt for Autonomous Drivingcited by 111
  22. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 104
  23. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  24. 2023Referring Multi-Object Trackingcited by 93
  25. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 89
  26. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  27. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 50
  28. 2021CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptionscited by 26
  29. 2022Ground then Navigate: Language-guided Navigation in Dynamic Scenescited by 20
  30. 2022ALERT: Adapting Language Models to Reasoning Taskscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 25; 2025: 56; 2026: 41.

123 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 260
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  3. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  4. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 146
  5. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 143
  6. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 92
  7. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  8. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  9. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  10. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  11. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  12. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 52
38 more
  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  3. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 41
  4. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  5. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  6. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 38
  7. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 36
  8. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understandingcited by 36
  9. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  10. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 33
  11. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  12. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  13. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  14. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  15. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 27
  16. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  17. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 25
  18. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  19. 2025Black-Box Adversarial Attack on Vision Language Models for Autonomous Drivingcited by 24
  20. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  21. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  22. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
  23. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  24. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 19
  25. 2024Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typographycited by 18
  26. 2025DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Modelscited by 17
  27. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  28. 2024Driving Style Alignment for LLM-powered Driver Agentcited by 15
  29. 2024On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validationcited by 15
  30. 2025InsightDrive: Insight Scene Representation for End-to-End Autonomous Drivingcited by 14
  31. 2025ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Drivingcited by 14
  32. 2024Large Language Models Powered Context-aware Motion Prediction in Autonomous Drivingcited by 13
  33. 2024VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detectioncited by 13
  34. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  35. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  36. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  37. 2025AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Drivingcited by 10
  38. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 10

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models
  2. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
  3. 2026Drive-P2D: A Progressive Perception-to-Decision Benchmark for VLMs in Autonomous Driving
  4. 2026Outcome-Guided Distillation: A Teacher-Student Framework to Advance VLM Reasoning in Autonomous Driving
  5. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
  6. 2025DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving
  7. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
  8. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views