mlsanityData to 7 Oct 2026

Multimodal End-to-End Autonomous Driving

arXiv 1906.03199 ↗First posted 7 Jun 2019cs.CVcited by 110

Not on the map. Why?

Cited by 42 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A crucial component of an autonomous vehicle (AV) is the artificial intelligence (AI) is able to drive towards a desired destination. Today, there are different paradigms addressing the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks such as perception and maneuver planning and control. On the other hand, we find end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals. The later are relatively less studied, but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to-end autonomous driving. So far, most proposals relying on this paradigm assume RGB images as input sensor data. However, AVs will not be equipped only with cameras, but also with active sensors providing accurate depth information (e.g., LiDARs). Accordingly, this paper analyses whether combining RGB and depth modalities, i.e. using RGBD data, produces better end-to-end AI drivers than relying on a single modality. We consider multimodality based on early, mid and late fusion schemes, both in multisensory and single-sensor (monocular depth estimation) settings. Using the CARLA simulator and conditional imitation learning (CIL), we show how, indeed, early fusion multimodality outperforms single-modality.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 16; 2023: 25; 2024: 22; 2025: 25; 2026: 8.

110 papers.

Sort:

By area:

  1. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  2. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  3. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  5. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  6. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  7. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  8. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  9. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  10. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  11. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
  12. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
38 more
  1. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  2. 2023Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Reviewcited by 57
  3. 2024HIMap: HybrId Representation Learning for End-to-end Vectorized HD Map Constructioncited by 48
  4. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 48
  5. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  6. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Modelcited by 44
  7. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  8. 2022Probabilistic Compositional Embeddings for Multimodal Image Retrievalcited by 29
  9. 2025Understanding the Emergence of Multimodal Representation Alignmentcited by 28
  10. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 27
  11. 2021Efficient and Robust LiDAR-Based End-to-End Navigationcited by 22
  12. 2021MASS: Multi-Attentional Semantic Segmentation of LiDAR Data for Dense Top-View Understandingcited by 22
  13. 2022Self-attention fusion for audiovisual emotion recognition with incomplete datacited by 22
  14. 2022End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agentcited by 21
  15. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  16. 2021Learning Interactive Driving Policies via Data-driven Simulationcited by 17
  17. 2024All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmarkcited by 14
  18. 2025Aligning Multimodal Representations through an Information Bottleneckcited by 12
  19. 2022Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Drivingcited by 10
  20. 2024MGMapNet: Multi-Granularity Representation Learning for End-to-End Vectorized HD Map Constructioncited by 9
  21. 2023Scaling Vision-based End-to-End Driving with Multi-View Attention Learningcited by 8
  22. 2024Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learningcited by 8
  23. 2025HisTrackMap: Global Vectorized High-Definition Map Construction via History Map Trackingcited by 8
  24. 2022LiDAR-as-Camera for End-to-End Drivingcited by 7
  25. 2023Goal-Guided Transformer-Enabled Reinforcement Learning for Efficient Autonomous Navigationcited by 7
  26. 2023Communication Resources Constrained Hierarchical Federated Learning for End-to-End Autonomous Drivingcited by 7
  27. 2024M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Drivingcited by 7
  28. 2021Motion Planning for Autonomous Vehicles in the Presence of Uncertainty Using Reinforcement Learningcited by 6
  29. 2022Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrationscited by 6
  30. 2023Risk-Aware Reward Shaping of Reinforcement Learning Agents for Autonomous Drivingcited by 6
  31. 2025CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systemscited by 6
  32. 2021Improving Generalization of Transfer Learning Across Domains Using Spatio-Temporal Features in Autonomous Drivingcited by 5
  33. 2021Safe Imitation Learning on Real-Life Highway Data for Human-like Autonomous Drivingcited by 5
  34. 2021A Versatile and Efficient Reinforcement Learning Framework for Autonomous Drivingcited by 5
  35. 2022Co-visual pattern augmented generative transformer learning for automobile geo-localizationcited by 5
  36. 2024LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Datacited by 5
  37. 2024VLASCD: A Visual Language Action Model for Simultaneous Chatting and Decision Makingcited by 5
  38. 2025FedEMA: Federated Exponential Moving Averaging with Negative Entropy Regularizer in Autonomous Drivingcited by 5

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models
  2. 2023End-to-end Autonomous Driving using Deep Learning: A Systematic Review
  3. 2026The Road Ahead in Autonomous Driving: The KITScenes Multimodal Dataset
  4. 2022MMFN: Multi-Modal-Fusion-Net for End-to-End Driving
  5. 2024EMMA: End-to-End Multimodal Model for Autonomous Driving
  6. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform
  7. 2025SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integration
  8. 2025Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles