# End-to-End Deep Learning of Lane Detection and Path Prediction for Real-Time Autonomous Driving

arXiv: https://arxiv.org/abs/2102.04738 · HTML: https://mlsanity.com/paper/2102.04738/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks](https://mlsanity.com/paper/1312.6229/) (2013)
- [Learning Lightweight Lane Detection CNNs by Self Attention Distillation](https://mlsanity.com/paper/1908.00821/) (2019)
- [Self-Driving Cars: A Survey](https://mlsanity.com/paper/1901.04407/) (2019)
- [Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection](https://mlsanity.com/paper/2010.12035/) (2020)
- [Towards End-to-End Lane Detection: an Instance Segmentation Approach](https://mlsanity.com/paper/1802.05591/) (2018)
- [3D-LaneNet: End-to-End 3D Multiple Lane Detection](https://mlsanity.com/paper/1811.10203/) (2018)
- [Fully Dense UNet for 2D Sparse Photoacoustic Tomography Artifact Removal](https://mlsanity.com/paper/1808.10848/) (2018)
- [VPGNet: Vanishing Point Guided Network for Lane and Road Marking Detection and Recognition](https://mlsanity.com/paper/1710.06288/) (2017)
- [Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks](https://mlsanity.com/paper/1903.02193/) (2019)
- [An Empirical Evaluation of Deep Learning on Highway Driving](https://mlsanity.com/paper/1504.01716/) (2015)
- [Learning a Driving Simulator](https://mlsanity.com/paper/1608.01230/) (2016)
- [Deep Learning and Control Algorithms of Direct Perception for Autonomous Driving](https://mlsanity.com/paper/1910.12031/) (2019)
- [SUPER: A Novel Lane Detection System](https://mlsanity.com/paper/2005.07277/) (2020)
- [Connection Sensitive Attention U-NET for Accurate Retinal Vessel Segmentation](https://mlsanity.com/paper/1903.05558/) (2019)
- [Advances in centerline estimation for autonomous lateral control](https://mlsanity.com/paper/2002.12685/) (2020)

## Built on by (most cited first, 16 of 16)

- [Harnessing Neuron Stability to Improve DNN Verification](https://mlsanity.com/paper/2401.14412/) (2024)
- [Recent Advancements in Deep Learning Applications and Methods for Autonomous Navigation: A Comprehensive Review](https://mlsanity.com/paper/2302.11089/) (2023)
- [Robust Lane Detection through Self Pre-training with Masked Sequential Autoencoders and Fine-tuning with Customized PolyLoss](https://mlsanity.com/paper/2305.17271/) (2023)
- [LCD-Net: A Lightweight Remote Sensing Change Detection Network Combining Feature Fusion and Gating Mechanism](https://mlsanity.com/paper/2410.11580/) (2024)
- [Advancing Autonomous Vehicle Intelligence: Deep Learning and Multimodal LLM for Traffic Sign Recognition and Robust Lane Detection](https://mlsanity.com/paper/2503.06313/) (2025)
- [Efficient Perception, Planning, and Control Algorithm for Vision-Based Automated Vehicles](https://mlsanity.com/paper/2209.07042/) (2022)
- [Contrastive Learning for Lane Detection via cross-similarity](https://mlsanity.com/paper/2308.08242/) (2023)
- [ComFe: An Interpretable Head for Vision Transformers](https://mlsanity.com/paper/2403.04125/) (2024)
- [Multi-task UNet architecture for end-to-end autonomous driving](https://mlsanity.com/paper/2112.08967/) (2021)
- [LOID: Lane Occlusion Inpainting and Detection for Enhanced Autonomous Driving Systems](https://mlsanity.com/paper/2408.09117/) (2024)
- [Uncertainty and Prediction Quality Estimation for Semantic Segmentation via Graph Neural Networks](https://mlsanity.com/paper/2409.11373/) (2024)
- [Real-Time Lane Detection via Efficient Feature Alignment and Covariance Optimization for Low-Power Embedded Systems](https://mlsanity.com/paper/2601.01696/) (2026)
- [All-Optical Segmentation via Diffractive Neural Networks for Autonomous Driving](https://mlsanity.com/paper/2602.07717/) (2026)
- [A Study on Real-time Object Detection using Deep Learning](https://mlsanity.com/paper/2602.15926/) (2026)
- [Multi-model approach for autonomous driving: A comprehensive study on traffic sign-, vehicle- and lane detection and behavioral cloning](https://mlsanity.com/paper/2603.09255/) (2026)
- [Lightweight Spatiotemporal Highway Lane Detection via 3D-ResNet and PINet with ROI-Aware Attention](https://mlsanity.com/paper/2604.02188/) (2026)

## Abstract (arXiv)

Inspired by the UNet architecture of semantic image segmentation, we propose a lightweight UNet using depthwise separable convolutions (DSUNet) for end-to-end learning of lane detection and path prediction (PP) in autonomous driving. We also design and integrate a PP algorithm with convolutional neural network (CNN) to form a simulation model (CNN-PP) that can be used to assess CNN's performance qualitatively, quantitatively, and dynamically in a host agent car driving along with other agents all in a real-time autonomous manner. DSUNet is 5.16x lighter in model size and 1.61x faster in inference than UNet. DSUNet-PP outperforms UNet-PP in mean average errors of predicted curvature and lateral offset for path planning in dynamic simulation. DSUNet-PP outperforms a modified UNet in lateral error, which is tested in a real car on real road. These results show that DSUNet is efficient and effective for lane detection and path prediction in autonomous driving.