# Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding

arXiv: https://arxiv.org/abs/2005.09202 · HTML: https://mlsanity.com/paper/2005.09202/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2020.

## Built on by (most cited first, 30 of 42)

- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving](https://mlsanity.com/paper/2303.05760/) (2023)
- [VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision](https://mlsanity.com/paper/2412.14446/) (2024)
- [ADAPT: Action-aware Driving Caption Transformer](https://mlsanity.com/paper/2302.00673/) (2023)
- [Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving](https://mlsanity.com/paper/2207.10422/) (2022)
- [Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving](https://mlsanity.com/paper/2103.10690/) (2021)
- [Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learning](https://mlsanity.com/paper/2010.03118/) (2020)
- [Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving](https://mlsanity.com/paper/2212.08787/) (2022)
- [Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving](https://mlsanity.com/paper/2102.09243/) (2021)
- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent](https://mlsanity.com/paper/2204.05513/) (2022)
- [Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving](https://mlsanity.com/paper/2104.07246/) (2021)
- [Bayesian Imitation Learning for End-to-End Mobile Manipulation](https://mlsanity.com/paper/2202.07600/) (2022)
- [Multi-modal Fusion Technology based on Vehicle Information: A Survey](https://mlsanity.com/paper/2211.06080/) (2022)
- [Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2407.13642/) (2024)
- [Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Driving](https://mlsanity.com/paper/2302.03939/) (2023)
- [A Versatile and Efficient Reinforcement Learning Framework for Autonomous Driving](https://mlsanity.com/paper/2110.11573/) (2021)
- [INSIGHT: Enhancing Autonomous Driving Safety through Vision-Language Models on Context-Aware Hazard Detection and Edge Case Evaluation](https://mlsanity.com/paper/2502.00262/) (2025)
- [Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems](https://mlsanity.com/paper/2104.04000/) (2021)
- [Interpretable Decision-Making for End-to-End Autonomous Driving](https://mlsanity.com/paper/2508.18898/) (2025)
- [Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Driving](https://mlsanity.com/paper/2106.12194/) (2021)
- [DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environments](https://mlsanity.com/paper/2207.09934/) (2022)
- [Sensor Fusion by Spatial Encoding for Autonomous Driving](https://mlsanity.com/paper/2308.10707/) (2023)
- [Self-Supervised Interpretable End-to-End Learning via Latent Functional Modularity](https://mlsanity.com/paper/2403.18947/) (2024)
- [A Threefold Review on Deep Semantic Segmentation: Efficiency-oriented, Temporal and Depth-aware design](https://mlsanity.com/paper/2303.04315/) (2023)
- [DeepIPCv2: LiDAR-powered Robust Environmental Perception and Navigational Control for Autonomous Vehicle](https://mlsanity.com/paper/2307.06647/) (2023)
- [Yaw-Guided Imitation Learning for Autonomous Driving in Urban Environments](https://mlsanity.com/paper/2111.06017/) (2021)
- [Towards Multimodal Multitask Scene Understanding Models for Indoor Mobile Agents](https://mlsanity.com/paper/2209.13156/) (2022)

## Abstract (arXiv)

This study aims to improve the performance and generalization capability of end-to-end autonomous driving with scene understanding leveraging deep learning and multimodal sensor fusion techniques. The designed end-to-end deep neural network takes as input the visual image and associated depth information in an early fusion level and outputs the pixel-wise semantic segmentation as scene understanding and vehicle control commands concurrently. The end-to-end deep learning-based autonomous driving model is tested in high-fidelity simulated urban driving conditions and compared with the benchmark of CoRL2017 and NoCrash. The testing results show that the proposed approach is of better performance and generalization ability, achieving a 100% success rate in static navigation tasks in both training and unobserved situations, as well as better success rates in other tasks than the prior models. A further ablation study shows that the model with the removal of multimodal sensor fusion or scene understanding pales in the new environment because of the false perception. The results verify that the performance of our model is improved by the synergy of multimodal sensor fusion with scene understanding subtask, demonstrating the feasibility and effectiveness of the developed deep neural network with multimodal sensor fusion.