# End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners

arXiv: https://arxiv.org/abs/1803.10158 · HTML: https://mlsanity.com/paper/1803.10158/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2018.

## Built on by (most cited first, 27 of 27)

- [End-to-End Urban Driving by Imitating a Reinforcement Learning Coach](https://mlsanity.com/paper/2108.08265/) (2021)
- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control](https://mlsanity.com/paper/2412.11198/) (2024)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterization](https://mlsanity.com/paper/2103.01882/) (2021)
- [Artificial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data](https://mlsanity.com/paper/2210.13462/) (2022)
- [Unsupervised Model Adaptation for Source-free Segmentation of Medical Images](https://mlsanity.com/paper/2211.00807/) (2022)
- [Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs](https://mlsanity.com/paper/2404.04869/) (2024)
- [Domain Adaptation for the Segmentation of Confidential Medical Images](https://mlsanity.com/paper/2101.00522/) (2021)
- [Enhancing Online Road Network Perception and Reasoning with Standard Definition Maps](https://mlsanity.com/paper/2408.01471/) (2024)
- [ShareVerse: Multi-Agent Consistent Video Generation for Shared World Modeling](https://mlsanity.com/paper/2603.02697/) (2026)
- [Driving through the Concept Gridlock: Unraveling Explainability Bottlenecks in Automated Driving](https://mlsanity.com/paper/2310.16639/) (2023)
- [SAGE: A Split-Architecture Methodology for Efficient End-to-End Autonomous Vehicle Control](https://mlsanity.com/paper/2107.10895/) (2021)
- [Towards End-to-End Deep Learning for Autonomous Racing: On Data Collection and a Unified Architecture for Steering and Throttle Prediction](https://mlsanity.com/paper/2105.01799/) (2021)
- [End-to-end Autonomous Driving using Deep Learning: A Systematic Review](https://mlsanity.com/paper/2311.18636/) (2023)
- [Deep Visual Navigation under Partial Observability](https://mlsanity.com/paper/2109.07752/) (2021)
- [Adversarial Mixture Density Networks: Learning to Drive Safely from Collision Data](https://mlsanity.com/paper/2107.04485/) (2021)
- [Conquering Ghosts: Relation Learning for Information Reliability Representation and End-to-End Robust Navigation](https://mlsanity.com/paper/2203.09952/) (2022)
- [MaskFuser: Masked Fusion of Joint Multi-Modal Tokenization for End-to-End Autonomous Driving](https://mlsanity.com/paper/2405.07573/) (2024)
- [A Survey on Datasets for Decision-making of Autonomous Vehicle](https://mlsanity.com/paper/2306.16784/) (2023)
- [Combating the effects of speed and delays in end-to-end self-driving](https://mlsanity.com/paper/2312.06670/) (2023)
- [Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages](https://mlsanity.com/paper/2103.09726/) (2021)
- [Yaw-Guided Imitation Learning for Autonomous Driving in Urban Environments](https://mlsanity.com/paper/2111.06017/) (2021)
- [Robust AI Driving Strategy for Autonomous Vehicles](https://mlsanity.com/paper/2207.07829/) (2022)
- [Geo-Context Aware Study of Vision-Based Autonomous Driving Models and Spatial Video Data](https://mlsanity.com/paper/2109.10895/) (2021)

## Abstract (arXiv)

For human drivers, having rear and side-view mirrors is vital for safe driving. They deliver a more complete view of what is happening around the car. Human drivers also heavily exploit their mental map for navigation. Nonetheless, several methods have been published that learn driving models with only a front-facing camera and without a route planner. This lack of information renders the self-driving task quite intractable. We investigate the problem in a more realistic setting, which consists of a surround-view camera system with eight cameras, a route planner, and a CAN bus reader. In particular, we develop a sensor setup that provides data for a 360-degree view of the area surrounding the vehicle, the driving route to the destination, and low-level driving maneuvers (e.g. steering angle and speed) by human drivers. With such a sensor setup we collect a new driving dataset, covering diverse driving scenarios and varying weather/illumination conditions. Finally, we learn a novel driving model by integrating information from the surround-view cameras and the route planner. Two route planners are exploited: 1) by representing the planned routes on OpenStreetMap as a stack of GPS coordinates, and 2) by rendering the planned routes on TomTom Go Mobile and recording the progression into a video. Our experiments show that: 1) 360-degree surround-view cameras help avoid failures made with a single front-view camera, in particular for city driving and intersection scenarios; and 2) route planners help the driving task significantly, especially for steering angle prediction.