mlsanityData to 7 Oct 2026

Unleash the Potential of Image Branch for Cross-modal 3D Object Detection

arXiv 2301.09077 ↗First posted 22 Jan 2023cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

To achieve reliable and precise scene understanding, autonomous vehicles typically incorporate multiple sensing modalities to capitalize on their complementary attributes. However, existing cross-modal 3D detectors do not fully utilize the image domain information to address the bottleneck issues of the LiDAR-based detectors. This paper presents a new cross-modal 3D object detector, namely UPIDet, which aims to unleash the potential of the image branch from two aspects. First, UPIDet introduces a new 2D auxiliary task called normalized local coordinate map estimation. This approach enables the learning of local spatial-aware features from the image modality to supplement sparse point clouds. Second, we discover that the representational capability of the point cloud backbone can be enhanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinct and effective point-to-pixel module. Extensive experiments and ablation studies validate the effectiveness of our method. Notably, we achieved the top rank in the highly competitive cyclist class of the KITTI benchmark at the time of submission. The source code is available at https://github.com/Eaphan/UPIDet.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 22 Jan 2023.

Builds on

51 of 60 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2016Pyramid Scene Parsing Networkcited by 2,472
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  5. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  6. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  7. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  8. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  9. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  10. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  11. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  12. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
38 more
  1. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  2. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  3. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  4. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  5. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  6. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  7. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 409
  8. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 366
  9. 2020Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloudcited by 335
  10. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  11. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 325
  12. 2019STD: Sparse-to-Dense 3D Object Detector for Point Cloudcited by 282
  13. 20163D Bounding Box Estimation Using Deep Learning and Geometrycited by 277
  14. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  15. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  16. 2019Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Drivingcited by 188
  17. 20203D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detectioncited by 176
  18. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  19. 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
  20. 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
  21. 2022Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Cloudscited by 131
  22. 2019Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detectioncited by 124
  23. 2018ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shapecited by 123
  24. 2021SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloudcited by 119
  25. 2019TANet: Robust 3D Object Detection from Point Clouds with Triple Attentioncited by 108
  26. 2022BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereocited by 89
  27. 2019Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspotscited by 84
  28. 2021MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagationcited by 82
  29. 2022Sparse Fuse Dense: Towards High Quality 3D Detection with Depth Completioncited by 82
  30. 2020End-to-End Pseudo-LiDAR for Image-Based 3D Object Detectioncited by 80
  31. 2022Point Density-Aware Voxels for LiDAR 3D Object Detectioncited by 70
  32. 2021CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimationcited by 56
  33. 2022SASA: Semantics-Augmented Set Abstraction for Point-based 3D Object Detectioncited by 52
  34. 2021EPNet++: Cascade Bi-directional Fusion for Multi-Modal 3D Object Detectioncited by 49
  35. 2022A Unified Query-based Paradigm for Point Cloud Understandingcited by 32
  36. 2022CAT-Det: Contrastively Augmented Transformer for Multi-modal 3D Object Detectioncited by 32
  37. 2022GLENet: Boosting 3D Object Detectors with Generative Label Uncertainty Estimationcited by 24
  38. 2022A Comprehensive Study of the Robustness for LiDAR-based 3D Object Detectors against Adversarial Attackscited by 22

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 2; 2026: 1.

11 papers.

  1. 2023Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detectioncited by 8
  2. 2024Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibrationcited by 7
  3. 2024Unsupervised 3D Point Cloud Completion via Multi-view Adversarial Learningcited by 6
  4. 2024Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Modelscited by 5
  5. 2025Reflectance Prediction-based Knowledge Distillation for Robust 3D Object Detection in Compressed Point Cloudscited by 3
  6. 2024Boosting 3D Object Detection with Semantic-Aware Multi-Branch Frameworkcited by 1
  7. 2025All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehiclescited by 1
  8. 2023SurfaceAug: Closing the Gap in Multimodal Ground Truth Samplingcited by 0
  9. 2024Hybrid Pooling and Convolutional Network for Improving Accuracy and Training Convergence Speed in Object Detectioncited by 0
  10. 2024Image-Guided Semantic Pseudo-LiDAR Point Generation for 3D Object Detectioncited by 0
  11. 2026Point Virtual Transformercited by 0