# Multi-task UNet architecture for end-to-end autonomous driving

arXiv: https://arxiv.org/abs/2112.08967 · HTML: https://mlsanity.com/paper/2112.08967/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [Self-Driving Cars: A Survey](https://mlsanity.com/paper/1901.04407/) (2019)
- [A Survey of End-to-End Driving: Architectures and Training Methods](https://mlsanity.com/paper/2003.06404/) (2020)
- [MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving](https://mlsanity.com/paper/1612.07695/) (2016)
- [Key Points Estimation and Point Instance Segmentation Approach for Lane Detection](https://mlsanity.com/paper/2002.06604/) (2020)
- [Conditional Affordance Learning for Driving in Urban Environments](https://mlsanity.com/paper/1806.06498/) (2018)
- [Y-Net: Joint Segmentation and Classification for Diagnosis of Breast Biopsy Images](https://mlsanity.com/paper/1806.01313/) (2018)
- [An Empirical Evaluation of Deep Learning on Highway Driving](https://mlsanity.com/paper/1504.01716/) (2015)
- [Robust Guarantees for Perception-Based Control](https://mlsanity.com/paper/1907.03680/) (2019)
- [Reinforcement Learning and Deep Learning based Lateral Control for Autonomous Driving](https://mlsanity.com/paper/1810.12778/) (2018)
- [End-to-End Deep Learning of Lane Detection and Path Prediction for Real-Time Autonomous Driving](https://mlsanity.com/paper/2102.04738/) (2021)
- [Deep Learning and Control Algorithms of Direct Perception for Autonomous Driving](https://mlsanity.com/paper/1910.12031/) (2019)
- [Control Strategies for Autonomous Vehicles](https://mlsanity.com/paper/2011.08729/) (2020)
- [Advances in centerline estimation for autonomous lateral control](https://mlsanity.com/paper/2002.12685/) (2020)

## Built on by (most cited first, 1 of 1)

- [Efficient Perception, Planning, and Control Algorithm for Vision-Based Automated Vehicles](https://mlsanity.com/paper/2209.07042/) (2022)

## Abstract (arXiv)

We propose an end-to-end driving model that integrates a multi-task UNet (MTUNet) architecture and control algorithms in a pipeline of data flow from a front camera through this model to driving decisions. It provides quantitative measures to evaluate the holistic, dynamic, and real-time performance of end-to-end driving systems and thus the safety and interpretability of MTUNet. The architecture consists of one segmentation, one regression, and two classification tasks for lane segmentation, path prediction, and vehicle controls. We present three variants of the architecture having different complexities, compare them on different tasks in four static measures for both single and multiple tasks, and then identify the best one by two additional dynamic measures in real-time simulation. Our results show that the performance of the proposed supervised learning model is comparable to that of a reinforcement learning model on curvy roads for the same task, which is not end-to-end but multi-module.