mlsanityData to 7 Oct 2026

Progressive Multi-Modal Fusion for Robust 3D Object Detection

arXiv 2410.07475 ↗First posted 9 Oct 2024cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Multi-sensor fusion is crucial for accurate 3D object detection in autonomous driving, with cameras and LiDAR being the most commonly used sensors. However, existing methods perform sensor fusion in a single view by projecting features from both modalities either in Bird's Eye View (BEV) or Perspective View (PV), thus sacrificing complementary information such as height or geometric proportions. To address this limitation, we propose ProFusion3D, a progressive fusion framework that combines features in both BEV and PV at both intermediate and object query levels. Our architecture hierarchically fuses local and global features, enhancing the robustness of 3D object detection. Additionally, we introduce a self-supervised mask modeling pre-training strategy to improve multi-modal representation learning and data efficiency through three novel objectives. Extensive experiments on nuScenes and Argoverse2 datasets conclusively demonstrate the efficacy of ProFusion3D. Moreover, ProFusion3D is robust to sensor failure, demonstrating strong performance when only one modality is available.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 48 of its 56 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

48 of 56 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  8. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  9. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  11. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  12. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
36 more
  1. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  2. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  3. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  4. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  5. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  6. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  7. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  8. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  9. 2019Class-balanced Grouping and Sampling for Point Cloud 3D Object Detectioncited by 350
  10. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 340
  11. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  12. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 326
  13. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  14. 2023VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Trackingcited by 221
  15. 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
  16. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  17. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  18. 2022DeepInteraction: 3D Object Detection via Modality Interactioncited by 125
  19. 2020ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentationcited by 95
  20. 2022Fully Sparse 3D Object Detectioncited by 84
  21. 2020Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detectioncited by 78
  22. 2023UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representationcited by 74
  23. 2022LargeKernel3D: Scaling up Kernels in 3D Sparse CNNscited by 71
  24. 2023Cross Modal Transformer: Towards Fast and Robust 3D Object Detectioncited by 68
  25. 2022Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupancy Autoencoderscited by 57
  26. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
  27. 2023Fully Sparse Fusion for 3D Object Detectioncited by 29
  28. 2023Detecting As Labeling: Rethinking LiDAR-camera Fusion in 3D Object Detectioncited by 28
  29. 2022Perceiving the Invisible: Proposal-Free Amodal Panoptic Segmentationcited by 20
  30. 2024CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibrationcited by 15
  31. 2023Panoptic Out-of-Distribution Segmentationcited by 13
  32. 2022Neural Architecture Search for Dense Prediction Tasks in Computer Visioncited by 11
  33. 2023EvCenterNet: Uncertainty Estimation for Object Detection using Evidential Learningcited by 10
  34. 2024A Point-Based Approach to Efficient LiDAR Multi-Task Perceptioncited by 9
  35. 2022On Hyperbolic Embeddings in 2D Object Detectioncited by 6
  36. 2024Automatic Target-Less Camera-LiDAR Calibration From Motion and Deep Point Correspondencescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2025Label-Efficient LiDAR Semantic Segmentation with 2D-3D Vision Transformer Adapterscited by 8
  2. 2026ForecastOcc: Vision-based Semantic Occupancy Forecastingcited by 6
  3. 2025Open-Set LiDAR Panoptic Segmentation Guided by Uncertainty-Aware Learningcited by 5
  4. 2025Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Modelscited by 5
  5. 2026UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentationcited by 5
  6. 2026Latent Gaussian Splatting for 4D Panoptic Occupancy Trackingcited by 3
  7. 2026Leveraging Previous-Traversal Point Cloud Map Priors for Camera-Based 3D Object Detection and Trackingcited by 2
  8. 2026GhostPoint: Self-Supervised Representation Learning by Hallucinating Occluded LiDAR Structurecited by 1
  9. 2026Effort-Based Criticality Metrics for Evaluating 3D Perception Errors in Autonomous Drivingcited by 0
  10. 2026Leveraging Vision-Based Point Cloud Map Priors for Camera-Based 3D Object Detection and Online Vectorized HD Mappingcited by 0