# Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car

arXiv: https://arxiv.org/abs/1704.07911 · HTML: https://mlsanity.com/paper/1704.07911/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2017.

## Built on by (most cited first, 30 of 66)

- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model](https://mlsanity.com/paper/2402.10828/) (2024)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safety](https://mlsanity.com/paper/1903.00640/) (2019)
- [End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation](https://mlsanity.com/paper/2406.17680/) (2024)
- [CoSA: Scheduling by Constrained Optimization for Spatial Accelerators](https://mlsanity.com/paper/2105.01898/) (2021)
- [Uncertainty-Aware Data Aggregation for Deep Imitation Learning](https://mlsanity.com/paper/1905.02780/) (2019)
- [EmergencyNet: Efficient Aerial Image Classification for Drone-Based Emergency Monitoring Using Atrous Convolutional Feature Fusion](https://mlsanity.com/paper/2104.14006/) (2021)
- [Adversarial Patch Attacks and Defences in Vision-Based Tasks: A Survey](https://mlsanity.com/paper/2206.08304/) (2022)
- [Robotic Table Tennis: A Case Study into a High Speed Learning System](https://mlsanity.com/paper/2309.03315/) (2023)
- [Safe end-to-end imitation learning for model predictive control](https://mlsanity.com/paper/1803.10231/) (2018)
- [MDPFuzz: Testing Models Solving Markov Decision Processes](https://mlsanity.com/paper/2112.02807/) (2021)
- [Robust Behavioral Cloning for Autonomous Vehicles using End-to-End Imitation Learning](https://mlsanity.com/paper/2010.04767/) (2020)
- [Behavioral Research and Practical Models of Drivers' Attention](https://mlsanity.com/paper/2104.05677/) (2021)
- [Scaling Vision-based End-to-End Driving with Multi-View Attention Learning](https://mlsanity.com/paper/2302.03198/) (2023)
- [InfoDisent: Explainability of Image Classification Models by Information Disentanglement](https://mlsanity.com/paper/2409.10329/) (2024)
- [Machine Learning for Security in Vehicular Networks: A Comprehensive Survey](https://mlsanity.com/paper/2105.15035/) (2021)
- [End-to-end Autonomous Driving using Deep Learning: A Systematic Review](https://mlsanity.com/paper/2311.18636/) (2023)
- [Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Driving](https://mlsanity.com/paper/2403.12176/) (2024)
- [Accelerating Imitation Learning with Predictive Models](https://mlsanity.com/paper/1806.04642/) (2018)
- [SIAM: Chiplet-based Scalable In-Memory Acceleration with Mesh for Deep Neural Networks](https://mlsanity.com/paper/2108.08903/) (2021)
- [Open challenges for Machine Learning based Early Decision-Making research](https://mlsanity.com/paper/2204.13111/) (2022)
- [Dynamic Conditional Imitation Learning for Autonomous Driving](https://mlsanity.com/paper/2211.11579/) (2022)
- [Tackling Real-World Autonomous Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/2207.02162/) (2022)
- [Label Encoding for Regression Networks](https://mlsanity.com/paper/2212.01927/) (2022)
- [Integrated Decision and Control at Multi-Lane Intersections with Mixed Traffic Flow](https://mlsanity.com/paper/2108.13038/) (2021)
- [PerD: Perturbation Sensitivity-based Neural Trojan Detection Framework on NLP Applications](https://mlsanity.com/paper/2208.04943/) (2022)
- [Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillation](https://mlsanity.com/paper/2305.16532/) (2023)

## Abstract (arXiv)

As part of a complete software stack for autonomous driving, NVIDIA has created a neural-network-based system, known as PilotNet, which outputs steering angles given images of the road ahead. PilotNet is trained using road images paired with the steering angles generated by a human driving a data-collection car. It derives the necessary domain knowledge by observing human drivers. This eliminates the need for human engineers to anticipate what is important in an image and foresee all the necessary rules for safe driving. Road tests demonstrated that PilotNet can successfully perform lane keeping in a wide variety of driving conditions, regardless of whether lane markings are present or not. The goal of the work described here is to explain what PilotNet learns and how it makes its decisions. To this end we developed a method for determining which elements in the road image most influence PilotNet's steering decision. Results show that PilotNet indeed learns to recognize relevant objects on the road. In addition to learning the obvious features such as lane markings, edges of roads, and other cars, PilotNet learns more subtle features that would be hard to anticipate and program by engineers, for example, bushes lining the edge of the road and atypical vehicle classes.