# Two-stream convolutional networks for end-to-end learning of self-driving cars

arXiv: https://arxiv.org/abs/1811.05785 · HTML: https://mlsanity.com/paper/1811.05785/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2018.

## Built on by (most cited first, 3 of 3)

- [Real-time End-to-End Federated Learning: An Automotive Case Study](https://mlsanity.com/paper/2103.11879/) (2021)
- [LiDAR-as-Camera for End-to-End Driving](https://mlsanity.com/paper/2206.15170/) (2022)
- [Controlling Steering with Energy-Based Models](https://mlsanity.com/paper/2301.12264/) (2023)

## Abstract (arXiv)

We propose a methodology to extend the concept of Two-Stream Convolutional Networks to perform end-to-end learning for self-driving cars with temporal cues. The system has the ability to learn spatiotemporal features by simultaneously mapping raw images and pre-calculated optical flows directly to steering commands. Although optical flows encode temporal-rich information, we found that 2D-CNNs are prone to capturing features only as spatial representations. We show how the use of Multitask Learning favors the learning of temporal features via inductive transfer from a shared spatiotemporal representation. Preliminary results demonstrate a competitive improvement of 30% in prediction accuracy and stability compared to widely used regression methods trained on the Comma.ai dataset.