mlsanityData to 7 Oct 2026

Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

arXiv 2202.02703 ↗First posted 6 Feb 2022cs.CVcited by 73

Not on the map. Why?

Abstract (arXiv)

Multi-modal fusion is a fundamental task for the perception of an autonomous driving system, which has recently intrigued many researchers. However, achieving a rather good performance is not an easy task due to the noisy raw data, underutilized information, and the misalignment of multi-modal sensors. In this paper, we provide a literature review of the existing multi-modal-based methods for perception tasks in autonomous driving. Generally, we make a detailed analysis including over 50 papers leveraging perception sensors including LiDAR and camera trying to solve object detection and semantic segmentation tasks. Different from traditional fusion methodology for categorizing fusion models, we propose an innovative way that divides them into two major classes, four minor classes by a more reasonable taxonomy in the view of the fusion stage. Moreover, we dive deep into the current fusion methods, focusing on the remaining problems and open-up discussions on the potential research opportunities. In conclusion, what we expect to do in this paper is to present a new taxonomy of multi-modal fusion methods for the autonomous driving perception tasks and provoke thoughts of the fusion-based techniques in the future.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 75 of its 107 references that are in the map. First posted on arXiv 6 Feb 2022.

Builds on

75 of 107 references.

References · 75

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  5. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  6. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  7. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  8. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  9. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  10. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  11. 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 494
  12. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
38 more
  1. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 409
  2. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  3. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 325
  4. 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 313
  5. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  6. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  7. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  8. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  9. 2019Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Drivingcited by 188
  10. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Drivingcited by 186
  11. 20203D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detectioncited by 176
  12. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  13. 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
  14. 2019Stereo R-CNN based 3D Object Detection for Autonomous Drivingcited by 163
  15. 2019MVX-Net: Multimodal VoxelNet for 3D Object Detectioncited by 162
  16. 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
  17. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 144
  18. 2019WoodScape: A multi-task, multi-camera fisheye dataset for autonomous drivingcited by 142
  19. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  20. 2019Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detectioncited by 124
  21. 2020RELLIS-3D Dataset: Data, Benchmarks and Analysiscited by 118
  22. 2020Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Cloudscited by 116
  23. 2020Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Reviewcited by 114
  24. 2018TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agentscited by 111
  25. 2020Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Reviewcited by 107
  26. 2018Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoningcited by 102
  27. 2019The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenescited by 80
  28. 2020Towards Semantic Segmentation of Urban-Scale 3D Point Clouds: A Dataset, Benchmarks and Challengescited by 80
  29. 2019A*3D Dataset: Towards Autonomous Driving in Challenging Environmentscited by 64
  30. 2021Multi-Modal 3D Object Detection in Autonomous Driving: a Surveycited by 63
  31. 20163D Object Proposals using Stereo Imagery for Accurate Object Class Detectioncited by 58
  32. 2019PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-based Attentive Cont-conv Fusion Modulecited by 56
  33. 2019Sensor Fusion for Joint 3D Object Detection and Semantic Segmentationcited by 48
  34. 2018IPOD: Intensive Point-based Object Detector for Point Cloudcited by 46
  35. 2020P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understandingcited by 41
  36. 2018LaneNet: Real-Time Lane Detection Networks for Autonomous Drivingcited by 39
  37. 2019Complexer-YOLO: Real-Time 3D Object Detection and Tracking on Semantic Point Cloudscited by 35
  38. 2018RoarNet: A Robust 3D Object Detection based on RegiOn Approximation Refinementcited by 31

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 21; 2024: 15; 2025: 23; 2026: 12.

73 papers.

Sort:

By area:

  1. 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 128
  2. 2023LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusioncited by 80
  3. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  4. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 58
  5. 2023Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Reviewcited by 57
  6. 2023UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebasecited by 57
  7. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  8. 2023DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Cloudscited by 31
  9. 2023Vision-Based Traffic Accident Detection and Anticipation: A Surveycited by 31
  10. 2022HRFuser: A Multi-resolution Sensor Fusion Architecture for 2D Object Detectioncited by 30
  11. 2025DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentationcited by 30
  12. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
38 more
  1. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 29
  2. 2024EfficientGS: Streamlining Gaussian Splatting for Large-Scale High-Resolution Scene Representationcited by 27
  3. 2022HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Drivingcited by 22
  4. 2023A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Applicationcited by 21
  5. 2024Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentationcited by 21
  6. 2025SuperPC: A Single Diffusion Model for Point Cloud Completion, Upsampling, Denoising, and Colorizationcited by 20
  7. 2024MultiCorrupt: A Multi-Modal Robustness Dataset and Benchmark of LiDAR-Camera Fusion for 3D Object Detectioncited by 17
  8. 2023Vision-RADAR fusion for Robotics BEV Detections: A Surveycited by 14
  9. 2024Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusioncited by 9
  10. 2025Collaborative Perception Datasets for Autonomous Driving: A Reviewcited by 8
  11. 2023Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Drivingcited by 6
  12. 2025Veila: Panoramic LiDAR Generation from a Monocular RGB Imagecited by 5
  13. 2025ROD: RGB-Only Fast and Efficient Off-road Freespace Detectioncited by 5
  14. 20233D Multiple Object Tracking on Autonomous Driving: A Literature Reviewcited by 4
  15. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Modelscited by 4
  16. 2023Benchmarking Robustness of AI-Enabled Multi-sensor Fusion Systems: Challenges and Opportunitiescited by 3
  17. 2024GDTM: An Indoor Geospatial Tracking Dataset with Distributed Multimodal Sensorscited by 3
  18. 2025Bridging Spectral-wise and Multi-spectral Depth Estimation via Geometry-guided Contrastive Learningcited by 3
  19. 2026Post Fusion Bird's Eye View Feature Stabilization for Robust Multimodal 3D Detectioncited by 3
  20. 2023Towards a Robust Sensor Fusion Step for 3D Object Detection on Corrupted Datacited by 2
  21. 2023Autonomous Advanced Aerial Mobility -- An End-to-end Autonomy Framework for UAVs and Beyondcited by 2
  22. 2024Rethinking RGB-D Fusion for Semantic Segmentation in Surgical Datasetscited by 2
  23. 2024A Comprehensive Review of 3D Object Detection in Autonomous Driving: Technological Advances and Future Directionscited by 2
  24. 2024MMLF: Multi-modal Multi-class Late Fusion for Object Detection with Uncertainty Estimationcited by 2
  25. 2025Image Segmentation with Large Language Models: A Survey with Perspectives for Intelligent Transportation Systemscited by 2
  26. 2025Exploring Fusion Strategies for Multimodal Vision-Language Systemscited by 2
  27. 2025New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehiclescited by 2
  28. 2026AsyncBEV: Cross-modal Flow Alignment in Asynchronous 3D Object Detectioncited by 2
  29. 2023A survey on deep learning approaches for data integration in autonomous driving systemcited by 1
  30. 2023FGFusion: Fine-Grained Lidar-Camera Fusion for 3D Object Detectioncited by 1
  31. 2023Dual-Branch Reconstruction Network for Industrial Anomaly Detection with RGB-D Datacited by 1
  32. 2024DiffPoint: Single and Multi-view Point Cloud Reconstruction with ViT Based Diffusion Modelcited by 1
  33. 2024Unified End-to-End V2X Cooperative Autonomous Drivingcited by 1
  34. 2024t-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous Drivingcited by 1
  35. 2025MESC-3D:Mining Effective Semantic Cues for 3D Reconstruction from a Single Imagecited by 1
  36. 2025HexPlane Representation for 3D Semantic Scene Understandingcited by 1
  37. 2025Dream-IF: Dynamic Relative EnhAnceMent for Image Fusioncited by 1
  38. 2025Uncertainty-Weighted Fusion of Image and Synthetic Event for Video Anomaly Detectioncited by 1

50 most cited of 73 All 73 (JSON)