mlsanityData to 7 Oct 2026

FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving

arXiv 2308.01006 ↗First posted 2 Aug 2023cs.CV cs.AI cs.ROcited by 67

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

Building a multi-modality multi-task neural network toward accurate and robust performance is a de-facto standard in perception task of autonomous driving. However, leveraging such data from multiple sensors to jointly optimize the prediction and planning tasks remains largely unexplored. In this paper, we present FusionAD, to the best of our knowledge, the first unified framework that fuse the information from two most critical sensors, camera and LiDAR, goes beyond perception task. Concretely, we first build a transformer based multi-modality fusion network to effectively produce fusion based features. In constrast to camera-based end-to-end method UniAD, we then establish a fusion aided modality-aware prediction and status-aware planning modules, dubbed FMSPnP that take advantages of multi-modality features. We conduct extensive experiments on commonly used benchmark nuScenes dataset, our FusionAD achieves state-of-the-art performance and surpassing baselines on average 15% on perception tasks like detection and tracking, 10% on occupancy prediction accuracy, reducing prediction error from 0.708 to 0.389 in ADE score and reduces the collision rate from 0.31% to only 0.12%.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 41 of its 45 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

41 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning15 Jul
  2. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun
  3. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Driving19 Apr

Other references · 38

  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  3. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  4. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  5. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  6. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  7. 2022Planning-oriented Autonomous Drivingcited by 1,068
  8. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  9. 2016End to End Learning for Self-Driving Carscited by 1,000
  10. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  11. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  12. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
26 more
  1. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  2. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  3. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  4. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  5. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  6. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 340
  7. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  8. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  9. 2018Learning to Drive in a Daycited by 266
  10. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  11. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  12. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  13. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  14. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 177
  15. 2021Multimodal Virtual Point 3D Detectioncited by 166
  16. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  17. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  18. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  19. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  20. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  21. 2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 99
  22. 2022Differentiable Raycasting for Self-supervised Occupancy Forecastingcited by 83
  23. 2022StretchBEV: Stretching Future Instance Prediction Spatially and Temporallycited by 47
  24. 2023PowerBEV: A Powerful Yet Lightweight Framework for Instance Prediction in Bird's-Eye Viewcited by 24
  25. 2022Path-Aware Graph Attention for HD Maps in Motion Predictioncited by 23
  26. 2023CCIL: Context-conditioned imitation learning for urban drivingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 20; 2025: 31; 2026: 9.

67 papers.

Sort:

By area:

  1. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  2. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  3. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  4. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  5. 2025GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Drivingcited by 117
  6. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  7. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 91
  8. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  9. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  10. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  11. 2024SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Drivingcited by 46
  12. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
38 more
  1. 2023PPAD: Iterative Interactions of Prediction and Planning for End-to-end Autonomous Drivingcited by 33
  2. 2025AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Drivingcited by 32
  3. 2024GaussianAD: Gaussian-Centric End-to-End Autonomous Drivingcited by 31
  4. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  5. 2024GraphAD: Interaction Scene Graph for End-to-end Autonomous Drivingcited by 25
  6. 2023PointBeV: A Sparse Approach to BeV Predictionscited by 23
  7. 2025Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Drivingcited by 19
  8. 2024Hybrid-Prediction Integrated Planning for Autonomous Drivingcited by 18
  9. 2025Learning Personalized Driving Styles via Reinforcement Learning from Human Feedbackcited by 16
  10. 2025GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Drivingcited by 16
  11. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  12. 2025UncAD: Towards Safe End-to-end Autonomous Driving via Online Map Uncertaintycited by 14
  13. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  14. 2025ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Drivingcited by 12
  15. 2025RealEngine: Simulating Autonomous Driving in Realistic Contextcited by 10
  16. 2025Enhancing Autonomous Driving Safety with Collision Scenario Integrationcited by 9
  17. 2024M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Drivingcited by 7
  18. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  19. 2024Joint Perception and Prediction for Autonomous Driving: A Surveycited by 7
  20. 2025Fully Unified Motion Planning for End-to-End Autonomous Drivingcited by 6
  21. 2025FocalAD: Local Motion Planning for End-to-End Autonomous Drivingcited by 6
  22. 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospectscited by 5
  23. 2025LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Drivingcited by 5
  24. 2025CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Modelcited by 5
  25. 2025OmniGen: Unified Multimodal Sensor Generation for Autonomous Drivingcited by 5
  26. 2023Achelous++: Power-Oriented Water-Surface Panoptic Perception Framework on Edge Devices based on Vision-Radar Fusion and Pruning of Heterogeneous Modalitiescited by 4
  27. 2024Pioneering SE(2)-Equivariant Trajectory Planning for Automated Drivingcited by 4
  28. 2024Hard Cases Detection in Motion Prediction by Vision-Language Foundation Modelscited by 4
  29. 2024LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperationcited by 4
  30. 2024Monocular Lane Detection Based on Deep Learning: A Surveycited by 4
  31. 2025CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Drivingcited by 4
  32. 2025Decoupling Scene Perception and Ego Status: A Multi-Context Fusion Approach for Enhanced Generalization in End-to-End Autonomous Drivingcited by 4
  33. 2024Depth Matters: Exploring Deep Interactions of RGB-D for Semantic Segmentation in Traffic Scenescited by 3
  34. 2024EgoFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Drivingcited by 3
  35. 2025Echo Planning for Autonomous Driving: From Current Observations to Future Trajectories and Backcited by 3
  36. 2025MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detectioncited by 3
  37. 2025UniLION: Towards Unified Autonomous Driving Model with Linear Group RNNscited by 3
  38. 2025ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Expertscited by 3

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles
  2. 2024DeepInteraction++: Multi-Modality Interaction for Autonomous Driving
  3. 2026MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving
  4. 2022MMFN: Multi-Modal-Fusion-Net for End-to-End Driving
  5. 2026MOJITO: Modal Joint Learning for Unified End-to-End Autonomous Driving
  6. 2025M3Net: Multimodal Multi-task Learning for 3D Detection, Segmentation, and Occupancy Prediction in Autonomous Driving
  7. 2025DiffE2E: Rethinking End-to-End Driving with a Hybrid Action Diffusion and Supervised Policy
  8. 2024MaskFuser: Masked Fusion of Joint Multi-Modal Tokenization for End-to-End Autonomous Driving