mlsanityData to 7 Oct 2026

MMFN: Multi-Modal-Fusion-Net for End-to-End Driving

arXiv 2207.00186 ↗First posted 1 Jul 2022cs.CV cs.ROcited by 17

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

Inspired by the fact that humans use diverse sensory organs to perceive the world, sensors with different modalities are deployed in end-to-end driving to obtain the global context of the 3D scene. In previous works, camera and LiDAR inputs are fused through transformers for better driving performance. These inputs are normally further interpreted as high-level map information to assist navigation tasks. Nevertheless, extracting useful information from the complex map input is challenging, for redundant information may mislead the agent and negatively affect driving performance. We propose a novel approach to efficiently extract features from vectorized High-Definition (HD) maps and utilize them in the end-to-end driving tasks. In addition, we design a new expert to further enhance the model performance by considering multi-road rules. Experimental results prove that both of the proposed improvements enable our agent to achieve superior performance compared with other methods.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 27 of its 32 references that are in the map. First posted on arXiv 1 Jul 2022.

Builds on

27 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Driving19 Apr
  2. 2021Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles16 Apr

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2017Graph Attention Networkscited by 8,854
  4. 2018YOLOv3: An Incremental Improvementcited by 2,652
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  7. 2016End to End Learning for Self-Driving Carscited by 1,000
  8. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  9. 2019Learning by Cheatingcited by 361
  10. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  11. 2021HDMapNet: An Online HD Map Construction and Evaluation Frameworkcited by 280
  12. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
13 more
  1. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  2. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  3. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  4. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  5. 2021Learning to drive from a world on railscited by 114
  6. 2021Ground-aware Monocular 3D Object Detection for Autonomous Drivingcited by 65
  7. 2021GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Drivingcited by 59
  8. 2021RoadMap: A Light-Weight Semantic Map for Visual Localization towards Autonomous Drivingcited by 34
  9. 2022Real-Time Trajectory Planning for Autonomous Driving with Gaussian Process and Incremental Refinementcited by 14
  10. 2021Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learningcited by 11
  11. 2021Greedy-Based Feature Selection for Efficient LiDAR SLAMcited by 9
  12. 2020A Software Architecture for Autonomous Vehicles: Team LRM-B Entry in the First CARLA Autonomous Driving Challengecited by 5
  13. 2020The Role of the Hercules Autonomous Vehicle During the COVID-19 Pandemic: An Autonomous Logistic Vehicle for Contactless Goods Transportationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 5; 2025: 2; 2026: 1.

17 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  3. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  4. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  5. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  6. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  7. 2023Rethinking Imitation-based Planner for Autonomous Drivingcited by 25
  8. 2025UncAD: Towards Safe End-to-end Autonomous Driving via Online Map Uncertaintycited by 14
  9. 2024Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMscited by 13
  10. 2023Transformer-Based Autonomous Driving Models and Deployment-Oriented Compression: A Surveycited by 8
  11. 2023Testing learning-enabled cyber-physical systems with Large-Language Models: A Formal Approachcited by 5
  12. 2024A Survey of Vision Transformers in Autonomous Driving: Current Trends and Future Directionscited by 4
5 more
  1. 2024MaskFuser: Masked Fusion of Joint Multi-Modal Tokenization for End-to-End Autonomous Drivingcited by 3
  2. 2024Enhancing End-to-End Autonomous Driving Systems Through Synchronized Human Behavior Datacited by 3
  3. 2023A Generalized Multi-Modal Fusion Detection Frameworkcited by 2
  4. 2023PaaS: Planning as a Service for reactive driving in CARLA Leaderboardcited by 1
  5. 2026Herding End-to-End Autonomous Driving via Neuro-Symbolic Safety Guardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving
  2. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving
  3. 2025Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles
  4. 2025MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion
  5. 2019Multimodal End-to-End Autonomous Driving
  6. 2026The Road Ahead in Autonomous Driving: The KITScenes Multimodal Dataset
  7. 2022Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving
  8. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives