# End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent

arXiv: https://arxiv.org/abs/2204.05513 · HTML: https://mlsanity.com/paper/2204.05513/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 29 of its 60 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [Multi-Task Multi-Sensor Fusion for 3D Object Detection](https://mlsanity.com/paper/2012.12397/) (2020)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [PnPNet: End-to-End Perception and Prediction with Tracking in the Loop](https://mlsanity.com/paper/2005.14711/) (2020)
- [Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?](https://mlsanity.com/paper/2006.14911/) (2020)
- [A Survey of End-to-End Driving: Architectures and Training Methods](https://mlsanity.com/paper/2003.06404/) (2020)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review](https://mlsanity.com/paper/2004.05224/) (2020)
- [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020)
- [Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/1911.04175/) (2019)
- [Multi-task Learning with Attention for End-to-end Autonomous Driving](https://mlsanity.com/paper/2104.10753/) (2021)
- [Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding](https://mlsanity.com/paper/2005.09202/) (2020)
- [Dynamic Task Weighting Methods for Multi-task Networks in Autonomous Driving Systems](https://mlsanity.com/paper/2001.02223/) (2020)
- [Volumetric Capture of Humans with a Single RGBD Camera via Semi-Parametric Learning](https://mlsanity.com/paper/1905.12162/) (2019)
- [Sample Efficient Interactive End-to-End Deep Learning for Self-Driving Cars with Selective Multi-Class Safe Dataset Aggregation](https://mlsanity.com/paper/2007.14671/) (2020)
- [AutonoVi: Autonomous Vehicle Planning with Dynamic Maneuvers and Traffic Constraints](https://mlsanity.com/paper/1703.08561/) (2017)

## Built on by (most cited first, 21 of 21)

- [Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges](https://mlsanity.com/paper/2308.10407/) (2023)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [MonoDiffusion: Self-Supervised Monocular Depth Estimation Using Diffusion Model](https://mlsanity.com/paper/2311.07198/) (2023)
- [M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving](https://mlsanity.com/paper/2403.12552/) (2024)
- [Revisiting Deformable Convolution for Depth Completion](https://mlsanity.com/paper/2308.01905/) (2023)
- [MMCert: Provable Defense against Adversarial Attacks to Multi-modal Models](https://mlsanity.com/paper/2403.19080/) (2024)
- [Modality Translation for Object Detection Adaptation Without Forgetting Prior Knowledge](https://mlsanity.com/paper/2404.01492/) (2024)
- [Interpretable Decision-Making for End-to-End Autonomous Driving](https://mlsanity.com/paper/2508.18898/) (2025)
- [DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environments](https://mlsanity.com/paper/2207.09934/) (2022)
- [SynDiff-AD: Improving Semantic Segmentation and End-to-End Autonomous Driving with Synthetic Data from Latent Diffusion Models](https://mlsanity.com/paper/2411.16776/) (2024)
- [DeepIPCv2: LiDAR-powered Robust Environmental Perception and Navigational Control for Autonomous Vehicle](https://mlsanity.com/paper/2307.06647/) (2023)
- [What Matters to Enhance Traffic Rule Compliance of Imitation Learning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2309.07808/) (2023)
- [Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving](https://mlsanity.com/paper/2210.06758/) (2022)
- [LeTFuser: Light-weight End-to-end Transformer-Based Sensor Fusion for Autonomous Driving with Multi-Task Learning](https://mlsanity.com/paper/2310.13135/) (2023)
- [Seq-DeepIPC: Sequential Sensing for End-to-End Control in Legged Robot Navigation](https://mlsanity.com/paper/2510.23057/) (2025)
- [Improved monocular depth prediction using distance transform over pre-semantic contours with self-supervised neural networks](https://mlsanity.com/paper/2605.08320/) (2026)
- [Progression Cognition Reinforcement Learning with Prioritized Experience for Multi-Vehicle Pursuit](https://mlsanity.com/paper/2306.05016/) (2023)
- [SKGE-SWIN: End-To-End Autonomous Vehicle Waypoint Prediction and Navigation Using Skip Stage Swin Transformer](https://mlsanity.com/paper/2508.20762/) (2025)
- [MTA-RL: Robust Urban Driving via Multi-modal Transformer-based 3D Affordances and Reinforcement Learning](https://mlsanity.com/paper/2605.10177/) (2026)
- [DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance](https://mlsanity.com/paper/2606.01277/) (2026)
- [A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving](https://mlsanity.com/paper/2608.20890/) (2026)

## Abstract (arXiv)

Focusing on the task of point-to-point navigation for an autonomous driving vehicle, we propose a novel deep learning model trained with end-to-end and multi-task learning manners to perform both perception and control tasks simultaneously. The model is used to drive the ego vehicle safely by following a sequence of routes defined by the global planner. The perception part of the model is used to encode high-dimensional observation data provided by an RGBD camera while performing semantic segmentation, semantic depth cloud (SDC) mapping, and traffic light state and stop sign prediction. Then, the control part decodes the encoded features along with additional information provided by GPS and speedometer to predict waypoints that come with a latent feature space. Furthermore, two agents are employed to process these outputs and make a control policy that determines the level of steering, throttle, and brake as the final action. The model is evaluated on CARLA simulator with various scenarios made of normal-adversarial situations and different weathers to mimic real-world conditions. In addition, we do a comparative study with some recent models to justify the performance in multiple aspects of driving. Moreover, we also conduct an ablation study on SDC mapping and multi-agent to understand their roles and behavior. As a result, our model achieves the highest driving score even with fewer parameters and computation load. To support future studies, we share our codes at https://github.com/oskarnatan/end-to-end-driving.