# End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception

arXiv: https://arxiv.org/abs/1801.06734 · HTML: https://mlsanity.com/paper/1801.06734/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jan 2018.

## Built on by (most cited first, 27 of 27)

- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving](https://mlsanity.com/paper/2209.08953/) (2022)
- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [Multi-modal Fusion Technology based on Vehicle Information: A Survey](https://mlsanity.com/paper/2211.06080/) (2022)
- [Imitation Learning of Hierarchical Driving Model: from Continuous Intention to Continuous Trajectory](https://mlsanity.com/paper/2010.10393/) (2020)
- [Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving](https://mlsanity.com/paper/2303.01788/) (2023)
- [Towards End-to-End Deep Learning for Autonomous Racing: On Data Collection and a Unified Architecture for Steering and Throttle Prediction](https://mlsanity.com/paper/2105.01799/) (2021)
- [Deep Visual Navigation under Partial Observability](https://mlsanity.com/paper/2109.07752/) (2021)
- [Imitation Learning for End to End Vehicle Longitudinal Control with Forward Camera](https://mlsanity.com/paper/1812.05841/) (2018)
- [Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems](https://mlsanity.com/paper/2104.04000/) (2021)
- [An Exploratory Study of AI System Risk Assessment from the Lens of Data Distribution and Uncertainty](https://mlsanity.com/paper/2212.06828/) (2022)
- [Efficient Baselines for Motion Prediction in Autonomous Driving](https://mlsanity.com/paper/2309.03387/) (2023)
- [Addressing Limitations of State-Aware Imitation Learning for Autonomous Driving](https://mlsanity.com/paper/2310.20650/) (2023)
- [Affordance-based Reinforcement Learning for Urban Driving](https://mlsanity.com/paper/2101.05970/) (2021)
- [How To Not Drive: Learning Driving Constraints from Demonstration](https://mlsanity.com/paper/2110.00645/) (2021)
- [On the Choice of Data for Efficient Training and Validation of End-to-End Driving Models](https://mlsanity.com/paper/2206.00608/) (2022)
- [SwapTransformer: highway overtaking tactical planner model via imitation learning on OSHA dataset](https://mlsanity.com/paper/2401.01425/) (2024)
- [Evaluating Robustness over High Level Driving Instruction for Autonomous Driving](https://mlsanity.com/paper/2105.10014/) (2021)
- [Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time Series Forecasting](https://mlsanity.com/paper/2107.12674/) (2021)
- [Towards Multimodal Multitask Scene Understanding Models for Indoor Mobile Agents](https://mlsanity.com/paper/2209.13156/) (2022)
- [One-Shot Learning of Visual Path Navigation for Autonomous Vehicles](https://mlsanity.com/paper/2306.08865/) (2023)
- [LaksNet: an end-to-end deep learning model for self-driving cars in Udacity simulator](https://mlsanity.com/paper/2310.16103/) (2023)
- [Unified End-to-End V2X Cooperative Autonomous Driving](https://mlsanity.com/paper/2405.03971/) (2024)
- [Incorporating Orientations into End-to-end Driving Model for Steering Control](https://mlsanity.com/paper/2103.05846/) (2021)
- [A Concise Tiling Strategy for Preserving Spatial Context in Earth Observation Imagery](https://mlsanity.com/paper/2404.10927/) (2024)
- [BEVPose: Unveiling Scene Semantics through Pose-Guided Multi-Modal BEV Alignment](https://mlsanity.com/paper/2410.20969/) (2024)
- [A Survey on Deep Multi-Task Learning in Connected Autonomous Vehicles](https://mlsanity.com/paper/2508.00917/) (2025)

## Abstract (arXiv)

Convolutional Neural Networks (CNN) have been successfully applied to autonomous driving tasks, many in an end-to-end manner. Previous end-to-end steering control methods take an image or an image sequence as the input and directly predict the steering angle with CNN. Although single task learning on steering angles has reported good performances, the steering angle alone is not sufficient for vehicle control. In this work, we propose a multi-task learning framework to predict the steering angle and speed control simultaneously in an end-to-end manner. Since it is nontrivial to predict accurate speed values with only visual inputs, we first propose a network to predict discrete speed commands and steering angles with image sequences. Moreover, we propose a multi-modal multi-task network to predict speed values and steering angles by taking previous feedback speeds and visual recordings as inputs. Experiments are conducted on the public Udacity dataset and a newly collected SAIC dataset. Results show that the proposed model predicts steering angles and speed values accurately. Furthermore, we improve the failure data synthesis methods to solve the problem of error accumulation in real road tests.