mlsanityData to 14 Oct 2026

Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car

arXiv 1704.07911 ↗First posted 25 Apr 2017cs.CV cs.LG cs.NEcited by 66

Not on the map. Why?

Cited by 25 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As part of a complete software stack for autonomous driving, NVIDIA has created a neural-network-based system, known as PilotNet, which outputs steering angles given images of the road ahead. PilotNet is trained using road images paired with the steering angles generated by a human driving a data-collection car. It derives the necessary domain knowledge by observing human drivers. This eliminates the need for human engineers to anticipate what is important in an image and foresee all the necessary rules for safe driving. Road tests demonstrated that PilotNet can successfully perform lane keeping in a wide variety of driving conditions, regardless of whether lane markings are present or not. The goal of the work described here is to explain what PilotNet learns and how it makes its decisions. To this end we developed a method for determining which elements in the road image most influence PilotNet's steering decision. Results show that PilotNet indeed learns to recognize relevant objects on the road. In addition to learning the obvious features such as lane markings, edges of roads, and other cars, PilotNet learns more subtle features that would be hard to anticipate and program by engineers, for example, bushes lining the edge of the road and atypical vehicle classes.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 2; 2019: 3; 2020: 2; 2021: 20; 2022: 12; 2023: 8; 2024: 7; 2025: 4; 2026: 8.

66 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  2. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 92
  3. 2019Urban Driving with Conditional Imitation Learningcited by 86
  4. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  5. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 61
  6. 2019Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safetycited by 49
  7. 2024End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotationcited by 28
  8. 2021CoSA: Scheduling by Constrained Optimization for Spatial Acceleratorscited by 22
  9. 2019Uncertainty-Aware Data Aggregation for Deep Imitation Learningcited by 20
  10. 2021EmergencyNet: Efficient Aerial Image Classification for Drone-Based Emergency Monitoring Using Atrous Convolutional Feature Fusioncited by 18
  11. 2022Adversarial Patch Attacks and Defences in Vision-Based Tasks: A Surveycited by 17
  12. 2023Robotic Table Tennis: A Case Study into a High Speed Learning Systemcited by 13
38 more
  1. 2018Safe end-to-end imitation learning for model predictive controlcited by 12
  2. 2021MDPFuzz: Testing Models Solving Markov Decision Processescited by 12
  3. 2020Robust Behavioral Cloning for Autonomous Vehicles using End-to-End Imitation Learningcited by 11
  4. 2021Behavioral Research and Practical Models of Drivers' Attentioncited by 9
  5. 2023Scaling Vision-based End-to-End Driving with Multi-View Attention Learningcited by 8
  6. 2024InfoDisent: Explainability of Image Classification Models by Information Disentanglementcited by 8
  7. 2021Machine Learning for Security in Vehicular Networks: A Comprehensive Surveycited by 7
  8. 2023End-to-end Autonomous Driving using Deep Learning: A Systematic Reviewcited by 7
  9. 2024Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Drivingcited by 7
  10. 2018Accelerating Imitation Learning with Predictive Modelscited by 6
  11. 2021SIAM: Chiplet-based Scalable In-Memory Acceleration with Mesh for Deep Neural Networkscited by 6
  12. 2022Open challenges for Machine Learning based Early Decision-Making researchcited by 6
  13. 2022Dynamic Conditional Imitation Learning for Autonomous Drivingcited by 6
  14. 2022Tackling Real-World Autonomous Driving using Deep Reinforcement Learningcited by 5
  15. 2022Label Encoding for Regression Networkscited by 5
  16. 2021Integrated Decision and Control at Multi-Lane Intersections with Mixed Traffic Flowcited by 4
  17. 2022PerD: Perturbation Sensitivity-based Neural Trojan Detection Framework on NLP Applicationscited by 4
  18. 2023Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillationcited by 3
  19. 2024PLM-Net: Perception Latency Mitigation Network for Vision-Based Lateral Control of Autonomous Vehiclescited by 3
  20. 2025Extensive Exploration in Complex Traffic Scenarios using Hierarchical Reinforcement Learningcited by 3
  21. 2026Lost in Fog: Sensor Perturbations Expose Reasoning Fragility in Driving VLAscited by 3
  22. 2020Enhanced Behavioral Cloning Based self-driving Car Using Transfer Learningcited by 2
  23. 2021Affordance-based Reinforcement Learning for Urban Drivingcited by 2
  24. 2021Co-Imitation Learning without Expert Demonstrationcited by 2
  25. 2021Bayesian Optimization and Deep Learning forsteering wheel angle predictioncited by 2
  26. 2023Learning Label Encodings for Deep Regressioncited by 2
  27. 2021A Topological Approach for Motion Track Discriminationcited by 1
  28. 2021MPC-Friendly Commitments for Publicly Verifiable Covert Securitycited by 1
  29. 2021Autonomous Racing using a Hybrid Imitation-Reinforcement Learning Architecturecited by 1
  30. 2022Generalizability Analysis of Graph-based Trajectory Predictor with Vectorized Representationcited by 1
  31. 2023Safety of autonomous vehicles: A survey on Model-based vs. AI-based approachescited by 1
  32. 2024Human Insights Driven Latent Space for Different Driving Perspectives: A Unified Encoder for Efficient Multi-Task Inferencecited by 1
  33. 2025CARIL: Confidence-Aware Regression in Imitation Learning for Autonomous Drivingcited by 1
  34. 2026MVAdapt: Zero-Shot Multi-Vehicle Adaptation for End-to-End Autonomous Drivingcited by 1
  35. 2021PredCoin: Defense against Query-based Hard-label Attackcited by 0
  36. 2021Incorporating Orientations into End-to-end Driving Model for Steering Controlcited by 0
  37. 2021Convex Online Video Frame Subset Selection using Multiple Criteria for Data Efficient Autonomous Drivingcited by 0
  38. 2021Brain-Inspired Deep Imitation Learning for Autonomous Driving Systemscited by 0

50 most cited of 66 All 66 (JSON)