# C-CoT: Counterfactual Chain-of-Thought with Vision-Language Models for Safe Autonomous Driving

arXiv: https://arxiv.org/abs/2605.10744 · HTML: https://mlsanity.com/paper/2605.10744/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning.
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/)
- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)
- End-to-end autonomous driving / [Reasoning VLA and VLM driving policies](https://mlsanity.com/driving/reasoning-vla-and-vlm-driving-policies/)

## Builds on (most cited first, 27 of 27)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024)
- [OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model](https://mlsanity.com/paper/2503.23463/) (2025)
- [Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving](https://mlsanity.com/paper/2410.22313/) (2024)
- [OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning](https://mlsanity.com/paper/2504.04348/) (2025)
- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving](https://mlsanity.com/paper/2312.03661/) (2023)
- [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023)
- [Explanations in Autonomous Driving: A Survey](https://mlsanity.com/paper/2103.05154/) (2021)
- [DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving](https://mlsanity.com/paper/2304.01168/) (2023)
- [Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoning](https://mlsanity.com/paper/2512.24426/) (2025)
- [Occlusion-Aware Risk Assessment for Autonomous Driving in Urban Environments](https://mlsanity.com/paper/1809.04629/) (2018)
- [Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects](https://mlsanity.com/paper/2505.20223/) (2025)
- [Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features](https://mlsanity.com/paper/2504.00557/) (2025)
- [Safe Imitation Learning on Real-Life Highway Data for Human-like Autonomous Driving](https://mlsanity.com/paper/2110.04052/) (2021)
- [Game-Theoretic Modeling of Vehicle Unprotected Left Turns Considering Drivers' Bounded Rationality](https://mlsanity.com/paper/2507.03002/) (2025)
- [LIFT: Interpretable truck driving risk prediction with literature-informed fine-tuned LLMs](https://mlsanity.com/paper/2510.22333/) (2025)
- [BAP-SRL: Bayesian Adaptive Priority Safe Reinforcement Learning for Vehicle Motion Planning at Mixed Traffic Intersections](https://mlsanity.com/paper/2601.21679/) (2026)
- [MPC-based Imitation Learning for Safe and Human-like Autonomous Driving](https://mlsanity.com/paper/2206.12348/) (2022)

## Built on by (most cited first, 3 of 3)

- [SafeAlign-VLA: A Negative-Enhanced Safe Alignment Framework for Risk-Aware Autonomous Driving](https://mlsanity.com/paper/2605.19524/) (2026)
- [Post-Training in End-to-End Autonomous Driving](https://mlsanity.com/paper/2607.08072/) (2026)
- [CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning](https://mlsanity.com/paper/2609.35823/) (2026)

## Abstract (arXiv)

Safety-critical planning in complex environments, particularly at urban intersections, remains a fundamental challenge for autonomous driving. Existing methods, whether rule-based or data-driven, frequently struggle to capture complex scene semantics, infer potential risks, and make reliable decisions in rare, high-risk situations. While vision-language models (VLMs) offer promising approaches for safe decision-making in these environments, most current approaches lack reflective and causal reasoning, thereby limiting their overall robustness. To address this, we propose a counterfactual chain-of-thought (C-CoT) framework that leverages VLMs to decompose driving decisions into five sequential stages: scene description, critical object identification, risk prediction, counterfactual risk reasoning, and final action planning. Within the counterfactual reasoning stage, we introduce a structured meta-action evaluation tree to explicitly assess the potential consequences of alternative action combinations. This self-reflective reasoning establishes causal links between action choices and safety outcomes, improving robustness in long-tail and out-of-distribution scenarios. To validate our approach, we construct the DeepAccident-CCoT dataset based on the DeepAccident benchmark and fine-tune a Qwen2.5-VL (7B) model using low-rank adaptation. Our model achieves a risk prediction recall of 81.9%, reduces the collision rate to 3.52%, and lowers L2 error to 1.98 m. Ablation studies further confirm the critical role of counterfactual reasoning and the meta-action evaluation tree in enhancing safety and interpretability.