# Explainable deep learning improves human mental models of self-driving cars

arXiv: https://arxiv.org/abs/2411.18714 · HTML: https://mlsanity.com/paper/2411.18714/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 18 of its 69 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 18 of 18)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Self-Driving Cars: A Survey](https://mlsanity.com/paper/1901.04407/) (2019)
- [LingoQA: Visual Question Answering for Autonomous Driving](https://mlsanity.com/paper/2312.14115/) (2023)
- [Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving](https://mlsanity.com/paper/2403.04133/) (2024)
- [Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions](https://mlsanity.com/paper/2112.11561/) (2021)
- [Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review](https://mlsanity.com/paper/2402.10086/) (2024)
- [The Utility of Explainable AI in Ad Hoc Human-Machine Teaming](https://mlsanity.com/paper/2209.03943/) (2022)
- [OCTET: Object-aware Counterfactual Explanations](https://mlsanity.com/paper/2211.12380/) (2022)
- [How Case Based Reasoning Explained Neural Networks: An XAI Survey of Post-Hoc Explanation-by-Example in ANN-CBR Twins](https://mlsanity.com/paper/1905.07186/) (2019)
- [Concept Bottleneck Models Without Predefined Concepts](https://mlsanity.com/paper/2407.03921/) (2024)
- [Human-Vehicle Cooperation on Prediction-Level: Enhancing Automated Driving with Human Foresight](https://mlsanity.com/paper/2104.03019/) (2021)
- [Incorporating Explanations into Human-Machine Interfaces for Trust and Situation Awareness in Autonomous Vehicles](https://mlsanity.com/paper/2404.07383/) (2024)
- [Lab2Car: A Versatile Wrapper for Deploying Experimental Planners in Complex Real-world Environments](https://mlsanity.com/paper/2409.09523/) (2024)
- [Surprise: Youve got some explaining to do](https://mlsanity.com/paper/1308.2236/) (2013)

## Built on by (most cited first, 1 of 1)

- [Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations](https://mlsanity.com/paper/2510.17256/) (2025)

## Abstract (arXiv)

Self-driving cars increasingly rely on deep neural networks to achieve human-like driving. The opacity of such black-box planners makes it challenging to accurately anticipate when they will fail, with potentially catastrophic consequences. While research into interpreting these systems has surged, most of it is confined to simulations or toy setups due to the difficulty of real-world deployment, leaving the practical utility of such techniques unknown. Here, we introduce the Concept-Wrapper Network (CW-Net), a method for faithfully explaining the behavior of machine-learning-based planners that causally grounds their reasoning in human-interpretable concepts without sacrificing performance. We deploy CW-Net on a real self-driving car and show that the resulting explanations improve the human driver's mental model of the vehicle, allowing them to better predict its behavior, particularly in surprising situations. This demonstrates that explainable deep learning integrated into self-driving cars can be both understandable and useful in a realistic deployment setting. We anticipate our method could be applied to other safety-critical systems, such as autonomous drones and robotic surgeons, as well as to other architectures, such as end-to-end learning systems and vision-language-action models. Overall, our study establishes a deployment-validated pathway to interpretability for autonomous agents, which could help make them more transparent and safe.