Data to 7 Oct 2026

End-to-end Learning of Driving Models from Large-scale Video Datasets

arXiv 1612.01079 ↗First posted 4 Dec 2016cs.CVcited by 147

Not on the map. Why?

Cited by 50 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Robust perception-action models should be learned from training data with diverse visual appearances and realistic behaviors, yet current approaches to deep visuomotor policy learning have been generally limited to in-situ models learned from a single vehicle or a simulation environment. We advocate learning a generic vehicle motion model from large scale crowd-sourced video data, and develop an end-to-end trainable architecture for learning to predict a distribution over future vehicle egomotion from instantaneous monocular camera observations and previous vehicle state. Our model incorporates a novel FCN-LSTM architecture, which can be learned from large-scale crowd-sourced vehicle action data, and leverages available scene segmentation side tasks to improve performance under a privileged learning paradigm.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 3; 2020: 4; 2021: 39; 2022: 25; 2023: 29; 2024: 24; 2025: 13; 2026: 8.

147 papers.

Sort:

By area:

2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
2022COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehiclescited by 95
2021Long Short-Term Transformer for Online Action Detectioncited by 87
2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76

38 more

2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 59
2021Deep Federated Learning for Autonomous Drivingcited by 54
2021Multi-task Learning with Attention for End-to-end Autonomous Drivingcited by 49
2019Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safetycited by 48
2021Triangle Attack: A Query-efficient Decision-based Adversarial Attackcited by 48
2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
2025OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigationcited by 34
2019MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learningcited by 29
2021Deep Learning-Based Autonomous Driving Systems: A Survey of Attacks and Defensescited by 29
2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
2023Towards Knowledge-driven Autonomous Drivingcited by 29
2021Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Drivingcited by 26
2023Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Drivingcited by 26
2021Self-Supervised Multi-Object Tracking with Cross-Input Consistencycited by 25
2021Efficient and Robust LiDAR-Based End-to-End Navigationcited by 22
2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
2023FishEye8K: A Benchmark and Dataset for Fisheye Camera Object Detectioncited by 21
2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
2023Effective Human-AI Teams via Learned Natural Language Rules and Onboardingcited by 20
2021A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learningcited by 18
2021Multi-Exit Semantic Segmentation Networkscited by 18
2021Learning Interactive Driving Policies via Data-driven Simulationcited by 17
2021PSI: A Benchmark for Human Interpretation and Response in Traffic Interactionscited by 17
2022Towards Domain Generalization in Object Detectioncited by 16
2022SelfD: Self-Learning Large-Scale Driving Policies From the Webcited by 16
2021DQ-GAT: Towards Safe and Efficient Autonomous Driving with Deep Q-Learning and Graph Attention Networkscited by 13
2021Facial-Sketch Synthesis: A New Challengecited by 13
2022Bayesian Imitation Learning for End-to-End Mobile Manipulationcited by 13
2022Learning Robotic Navigation from Experience: Principles, Methods, and Recent Resultscited by 13
2023A Study of Situational Reasoning for Traffic Understandingcited by 13
2024IDD-X: A Multi-View Dataset for Ego-relative Important Object Localization and Explanation in Dense and Unstructured Trafficcited by 13
2018OIL: Observational Imitation Learningcited by 11
2020Robust Behavioral Cloning for Autonomous Vehicles using End-to-End Imitation Learningcited by 11
2023Hierarchical Generative Adversarial Imitation Learning with Mid-level Input Generation for Autonomous Driving on Urban Environmentscited by 11
2019Gaze Training by Modulated Dropout Improves Imitation Learningcited by 10
2020Episodic Self-Imitation Learning with Hindsightcited by 10
2021A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehiclescited by 10

50 most cited of 147 All 147 (JSON)