mlsanityData to 14 Oct 2026

Multi-Modal 3D Object Detection in Autonomous Driving: a Survey

arXiv 2106.12735 ↗First posted 24 Jun 2021cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

In this survey, we first introduce the background of popular sensors used for self-driving, their data properties, and the corresponding object detection algorithms. Next, we discuss existing datasets that can be used for evaluating multi-modal 3D object detection algorithms. Then we present a review of multi-modal fusion based 3D detection networks, taking a close look at their fusion stage, fusion input and fusion granularity, and how these design choices evolve with time and technology. After the review, we discuss open challenges as well as possible solutions. We hope that this survey can help researchers to get familiar with the field and embark on investigations in the area of multi-modal 3D object detection.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 120 of its 226 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

120 of 226 references.

References · 120

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2017Mask R-CNNcited by 7,277
  9. 2016Densely Connected Convolutional Networkscited by 6,720
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  12. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
38 more
  1. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  6. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  7. 2015Fast R-CNNcited by 2,933
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  10. 2017Non-local Neural Networkscited by 2,293
  11. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  13. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  14. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  15. 2014How transferable are features in deep neural networks?cited by 1,225
  16. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  17. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  18. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  19. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  20. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  21. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
  22. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  23. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  24. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  25. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 724
  26. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
  27. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  28. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  29. 2018Deep Visual Domain Adaptation: A Surveycited by 635
  30. 2018Pyramid Stereo Matching Networkcited by 489
  31. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
  32. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  33. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  34. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  35. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 410
  36. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 368
  37. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  38. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 342

50 most cited of 120 All 120 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 12; 2024: 24; 2025: 8; 2026: 8.

63 papers.

Sort:

By area:

  1. 2023MapTRv2: An End-to-End Framework for Online Vectorized HD Map Constructioncited by 193
  2. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  3. 2023LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusioncited by 80
  4. 2022Multi-modal Sensor Fusion for Auto Driving Perception: A Surveycited by 76
  5. 2022BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detectioncited by 66
  6. 2021VPFNet: Improving 3D Object Detection with Virtual Point based LiDAR and Stereo Data Fusioncited by 48
  7. 2024Stream Query Denoising for Vectorized HD Map Constructioncited by 43
  8. 2022TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learningcited by 37
  9. 2023GraphAlign: Enhancing Accurate Feature Alignment by Graph matching for Multi-Modal 3D Object Detectioncited by 34
  10. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 31
  11. 2023Bi-LRFusion: Bi-Directional LiDAR-Radar Fusion for 3D Dynamic Object Detectioncited by 28
  12. 2023Towards Domain Generalization for Multi-view 3D Object Detection in Bird-Eye-Viewcited by 27
38 more
  1. 2024Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformercited by 27
  2. 2022Homogeneous Multi-modal Feature Fusion and Interaction for 3D Object Detectioncited by 26
  3. 2024Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentationcited by 22
  4. 2024RoboFusion: Towards Robust Multi-Modal 3D Object Detection via SAMcited by 20
  5. 2024MultiCorrupt: A Multi-Modal Robustness Dataset and Benchmark of LiDAR-Camera Fusion for 3D Object Detectioncited by 17
  6. 2024MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Drivingcited by 14
  7. 2024MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selectioncited by 12
  8. 2025RENO: Real-Time Neural Compression for 3D LiDAR Point Cloudscited by 12
  9. 2022JPerceiver: Joint Perception Network for Depth, Pose and Layout Estimation in Driving Scenescited by 11
  10. 2022Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detectioncited by 11
  11. 2024LiRaFusion: Deep Adaptive LiDAR-Radar Fusion for 3D Object Detectioncited by 11
  12. 2024HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View Distillationcited by 11
  13. 2021Agent-Centric Relation Graph for Object Visual Navigationcited by 10
  14. 2023Radars for Autonomous Driving: A Review of Deep Learning Methods and Challengescited by 10
  15. 2023Deep Transfer Learning for Intelligent Vehicle Perception: a Surveycited by 10
  16. 2024Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testingcited by 10
  17. 2024Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillationcited by 10
  18. 2024RayFormer: Improving Query-Based Multi-Camera 3D Object Detection via Ray-Centric Strategiescited by 9
  19. 2024Harnessing Uncertainty-aware Bounding Boxes for Unsupervised 3D Object Detectioncited by 9
  20. 2023OA-DET3D: Embedding Object Awareness as a General Plug-in for Multi-Camera 3D Object Detectioncited by 7
  21. 2024NanoMVG: USV-Centric Low-Power Multi-Task Visual Grounding based on Prompt-Guided Camera and 4D mmWave Radarcited by 7
  22. 2024UniBEVFusion: Unified Radar-Vision BEVFusion for 3D Object Detectioncited by 7
  23. 2024Part-Whole Relational Fusion Towards Multi-Modal Scene Understandingcited by 6
  24. 2025MoE3D: Mixture of Experts meets Multi-Modal 3D Understandingcited by 6
  25. 2022PointSee: Image Enhances Point Cloudcited by 5
  26. 2024DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perceptioncited by 5
  27. 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospectscited by 5
  28. 20233D Multiple Object Tracking on Autonomous Driving: A Literature Reviewcited by 4
  29. 2024NeRF-Det++: Incorporating Semantic Cues and Perspective-aware Depth Supervision for Indoor Multi-View 3D Detectioncited by 4
  30. 2024PoIFusion: Multi-Modal 3D Object Detection via Fusion at Points of Interestcited by 4
  31. 2022Emerging Threats in Deep Learning-Based Autonomous Driving: A Comprehensive Surveycited by 3
  32. 2023Enhancing Novel Object Detection via Cooperative Foundational Modelscited by 2
  33. 2024A Structure-Aware Lane Graph Transformer Model for Vehicle Trajectory Predictioncited by 2
  34. 2024InScope: A New Real-world 3D Infrastructure-side Collaborative Perception Dataset for Open Traffic Scenarioscited by 2
  35. 2025Continual Test-Time Adaptation for Single Image Defocus Deblurring via Causal Siamese Networkscited by 2
  36. 2025A Multimodal Hybrid Late-Cascade Fusion Network for Enhanced 3D Object Detectioncited by 2
  37. 2025A Novel Multi-layer Task-centric and Data Quality Framework for Autonomous Drivingcited by 2
  38. 2026AsyncBEV: Cross-modal Flow Alignment in Asynchronous 3D Object Detectioncited by 2

50 most cited of 63 All 63 (JSON)