# Memory based neural networks for end-to-end autonomous driving

arXiv: https://arxiv.org/abs/2205.12124 · HTML: https://mlsanity.com/paper/2205.12124/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 27 Apr 2022.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 12 of 12)

- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car](https://mlsanity.com/paper/1704.07911/) (2017)
- [Computing Systems for Autonomous Driving: State-of-the-Art and Challenges](https://mlsanity.com/paper/2009.14349/) (2020)
- [Learning a Driving Simulator](https://mlsanity.com/paper/1608.01230/) (2016)
- [End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies](https://mlsanity.com/paper/1710.03804/) (2017)
- [DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning](https://mlsanity.com/paper/1911.01562/) (2019)
- [Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues](https://mlsanity.com/paper/1708.03798/) (2017)
- [Feature Analysis and Selection for Training an End-to-End Autonomous Vehicle Controller Using the Deep Learning Approach](https://mlsanity.com/paper/1703.09744/) (2017)
- [Bayesian Optimization and Deep Learning forsteering wheel angle prediction](https://mlsanity.com/paper/2110.13629/) (2021)

## Abstract (arXiv)

Recent works in end-to-end control for autonomous driving have investigated the use of vision-based exteroceptive perception. Inspired by such results, we propose a new end-to-end memory-based neural architecture for robot steering and throttle control. We describe and compare this architecture with previous approaches using fundamental error metrics (MAE, MSE) and several external metrics based on their performance on simulated test circuits. The presented work demonstrates the advantages of using internal memory for better generalization capabilities of the model and allowing it to drive in a broader amount of circuits/situations. We analyze the algorithm in a wide range of environments and conclude that the proposed pipeline is robust to varying camera configurations. All the present work, including datasets, network models architectures, weights, simulator, and comparison software, is open source and easy to replicate and extend.