# Explaining Autonomous Driving by Learning End-to-End Visual Attention

arXiv: https://arxiv.org/abs/2006.03347 · HTML: https://mlsanity.com/paper/2006.03347/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2020.

## Built on by (most cited first, 25 of 25)

- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Explanations in Autonomous Driving: A Survey](https://mlsanity.com/paper/2103.05154/) (2021)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions](https://mlsanity.com/paper/2112.11561/) (2021)
- [Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review](https://mlsanity.com/paper/2402.10086/) (2024)
- [Multi-task Learning with Attention for End-to-end Autonomous Driving](https://mlsanity.com/paper/2104.10753/) (2021)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Surrogate Model-Based Explainability Methods for Point Cloud NNs](https://mlsanity.com/paper/2107.13459/) (2021)
- [CADRE: A Cascade Deep Reinforcement Learning Framework for Vision-based Autonomous Urban Driving](https://mlsanity.com/paper/2202.08557/) (2022)
- [On Salience-Sensitive Sign Classification in Autonomous Vehicle Path Planning: Experimental Explorations with a Novel Dataset](https://mlsanity.com/paper/2112.00942/) (2021)
- [Behavioral Research and Practical Models of Drivers' Attention](https://mlsanity.com/paper/2104.05677/) (2021)
- [Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Driving](https://mlsanity.com/paper/2403.12176/) (2024)
- [Towards explainable artificial intelligence (XAI) for early anticipation of traffic accidents](https://mlsanity.com/paper/2108.00273/) (2021)
- [Advancing Explainable Autonomous Vehicle Systems: A Comprehensive Review and Research Roadmap](https://mlsanity.com/paper/2404.00019/) (2024)
- [Vision Transformer for Learning Driving Policies in Complex Multi-Agent Environments](https://mlsanity.com/paper/2109.06514/) (2021)
- [Improving Explainable Object-induced Model through Uncertainty for Automated Vehicles](https://mlsanity.com/paper/2402.15572/) (2024)
- [ALINA: Advanced Line Identification and Notation Algorithm](https://mlsanity.com/paper/2406.08775/) (2024)
- [Addressing Limitations of State-Aware Imitation Learning for Autonomous Driving](https://mlsanity.com/paper/2310.20650/) (2023)
- [Reason induced visual attention for explainable autonomous driving](https://mlsanity.com/paper/2110.07380/) (2021)
- [On the Choice of Data for Efficient Training and Validation of End-to-End Driving Models](https://mlsanity.com/paper/2206.00608/) (2022)
- [Can Attribution Predict Risk? From Multi-View Attribution to Planning Risk Signals in End-to-End Autonomous Driving](https://mlsanity.com/paper/2605.06264/) (2026)
- [Measuring the Impact of Scene Level Objects on Object Detection: Towards Quantitative Explanations of Detection Decisions](https://mlsanity.com/paper/2401.10790/) (2024)
- [Behavioral Cloning Models Reality Check for Autonomous Driving](https://mlsanity.com/paper/2409.07218/) (2024)
- [Runway vs. Taxiway: Challenges in Automated Line Identification and Notation Approaches](https://mlsanity.com/paper/2501.18494/) (2025)
- [Reinforcement Learning Enhancement Using Vector Semantic Representation and Symbolic Reasoning for Human-Centered Autonomous Emergency Braking](https://mlsanity.com/paper/2602.05079/) (2026)

## Abstract (arXiv)

Current deep learning based autonomous driving approaches yield impressive results also leading to in-production deployment in certain controlled scenarios. One of the most popular and fascinating approaches relies on learning vehicle controls directly from data perceived by sensors. This end-to-end learning paradigm can be applied both in classical supervised settings and using reinforcement learning. Nonetheless the main drawback of this approach as also in other learning problems is the lack of explainability. Indeed, a deep network will act as a black-box outputting predictions depending on previously seen driving patterns without giving any feedback on why such decisions were taken. While to obtain optimal performance it is not critical to obtain explainable outputs from a learned agent, especially in such a safety critical field, it is of paramount importance to understand how the network behaves. This is particularly relevant to interpret failures of such systems. In this work we propose to train an imitation learning based agent equipped with an attention model. The attention model allows us to understand what part of the image has been deemed most important. Interestingly, the use of attention also leads to superior performance in a standard benchmark using the CARLA driving simulator.