mlsanityData to 14 Oct 2026

Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention

arXiv 1703.10631 ↗First posted 30 Mar 2017cs.CV cs.LGcited by 107

Not on the map. Why?

Cited by 31 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep neural perception and control networks are likely to be a key component of self-driving vehicles. These models need to be explainable - they should provide easy-to-interpret rationales for their behavior - so that passengers, insurance companies, law enforcement, developers etc., can understand what triggered a particular behavior. Here we explore the use of visual explanations. These explanations take the form of real-time highlighted regions of an image that causally influence the network's output (steering control). Our approach is two-stage. In the first stage, we use a visual attention model to train a convolution network end-to-end from images to steering angle. The attention model highlights image regions that potentially influence the network's output. Some of these are true influences, but some are spurious. We then apply a causal filtering step to determine which input regions actually influence the output. This produces more succinct visual explanations and more accurately exposes the network's behavior. We demonstrate the effectiveness of our model on three datasets totaling 16 hours of driving. We first show that training with attention does not degrade the performance of the end-to-end network. Then we show that the network causally cues on a variety of features that are used by humans while driving.

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 29; 2022: 18; 2023: 25; 2024: 20; 2025: 8; 2026: 5.

107 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  4. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 436
  5. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  6. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 145
  7. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  8. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  9. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 92
  10. 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 91
  11. 2021Explanations in Autonomous Driving: A Surveycited by 81
  12. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
38 more
  1. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  2. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  3. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  4. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 50
  5. 2021Multi-task Learning with Attention for End-to-end Autonomous Drivingcited by 49
  6. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  7. 2021DRIVE: Deep Reinforced Accident Anticipation with Visual Explanationcited by 46
  8. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Modelcited by 45
  9. 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 44
  10. 2023Learning Support and Trivial Prototypes for Interpretable Image Classificationcited by 38
  11. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
  12. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 32
  13. 2019MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learningcited by 29
  14. 2022OCTET: Object-aware Counterfactual Explanationscited by 28
  15. 2021Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterizationcited by 22
  16. 2021Self-Interpretable Model with TransformationEquivariant Interpretationcited by 21
  17. 2025AVD2: Accident Video Diffusion for Accident Video Descriptioncited by 20
  18. 2022CADRE: A Cascade Deep Reinforcement Learning Framework for Vision-based Autonomous Urban Drivingcited by 16
  19. 2023Explaining Autonomous Driving Actions with Visual Question Answeringcited by 16
  20. 2021Alignment Attention by Matching Key and Query Distributionscited by 14
  21. 2023A Review of the Role of Causality in Developing Trustworthy AI Systemscited by 14
  22. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14
  23. 2022Unsupervised Model Adaptation for Source-free Segmentation of Medical Imagescited by 13
  24. 2024IDD-X: A Multi-View Dataset for Ego-relative Important Object Localization and Explanation in Dense and Unstructured Trafficcited by 13
  25. 2021Domain Adaptation for the Segmentation of Confidential Medical Imagescited by 12
  26. 2025The Dead Salmons of AI Interpretabilitycited by 12
  27. 2022Learning Interpretable, High-Performing Policies for Autonomous Drivingcited by 11
  28. 2023Mixture of Gaussian-distributed Prototypes with Generative Modelling for Interpretable and Trustworthy Image Recognitioncited by 11
  29. 2023Goal-Guided Transformer-Enabled Reinforcement Learning for Efficient Autonomous Navigationcited by 10
  30. 2025Towards Model Resistant to Transferable Adversarial Examples via Trigger Activationcited by 10
  31. 2021Behavioral Research and Practical Models of Drivers' Attentioncited by 9
  32. 2021DROID: Driver-centric Risk Object Identificationcited by 9
  33. 2021Development and testing of an image transformer for explainable autonomous driving systemscited by 9
  34. 2023Interpretability-Aware Vision Transformercited by 8
  35. 2025There Was Never a Bottleneck in Concept Bottleneck Modelscited by 8
  36. 2021Towards Safe, Explainable, and Regulated Autonomous Drivingcited by 7
  37. 2024Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Drivingcited by 7
  38. 2024Structured Gradient-based Interpretations via Norm-Regularized Adversarial Trainingcited by 7

50 most cited of 107 All 107 (JSON)