mlsanityData to 14 Oct 2026

End to End Learning for Self-Driving Cars

arXiv 1604.07316 ↗First posted 25 Apr 2016cs.CV cs.LG cs.NEcited by 1,009

Not on the map. Why?

Cited by 254 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands. This end-to-end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e.g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS).

Key facts. Cited by 1,009 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Apr 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 7; 2019: 20; 2020: 20; 2021: 224; 2022: 168; 2023: 163; 2024: 168; 2025: 138; 2026: 101.

1,009 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  3. 2022Planning-oriented Autonomous Drivingcited by 1,087
  4. 2018Behavioral Cloning from Observationcited by 492
  5. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  6. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  7. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  8. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  9. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
  10. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  11. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  12. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
38 more
  1. 2019Causal Confusion in Imitation Learningcited by 288
  2. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 282
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  4. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 266
  5. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 248
  6. 2021End-to-end Interpretable Neural Motion Plannercited by 242
  7. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 242
  8. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  9. 2021Improving Robustness using Generated Datacited by 234
  10. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  11. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  12. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 212
  13. 2021Improving Calibration for Long-Tailed Recognitioncited by 212
  14. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  15. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 194
  16. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 193
  17. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  18. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  19. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 169
  20. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  21. 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 168
  22. 2019Goal-conditioned Imitation Learningcited by 162
  23. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 160
  24. 2025Pseudo-Simulation for Autonomous Drivingcited by 150
  25. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 146
  26. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 141
  27. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 137
  28. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  29. 2022Frequency Domain Model Augmentation for Adversarial Attackcited by 130
  30. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 129
  31. 2022Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigationcited by 127
  32. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  33. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
  34. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  35. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  36. 2021Learning to drive from a world on railscited by 114
  37. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 114
  38. 2022VideoDex: Learning Dexterity from Internet Videoscited by 106

50 most cited of 1,009 All 1,009 (JSON)