mlsanityData to 14 Oct 2026

Multimodal Virtual Point 3D Detection

arXiv 2111.06881 ↗First posted 12 Nov 2021cs.CV cs.LG cs.ROcited by 166

Not on the map. Why?

Abstract (arXiv)

Lidar-based sensing drives current autonomous vehicles. Despite rapid progress, current Lidar sensors still lag two decades behind traditional color cameras in terms of resolution and cost. For autonomous driving, this means that large objects close to the sensors are easily visible, but far-away or small objects comprise only one measurement or two. This is an issue, especially when these objects turn out to be driving hazards. On the other hand, these same objects are clearly visible in onboard RGB sensors. In this work, we present an approach to seamlessly fuse RGB sensors into Lidar-based 3D recognition. Our approach takes a set of 2D detections to generate dense 3D virtual points to augment an otherwise sparse 3D point cloud. These virtual points naturally integrate into any standard Lidar-based 3D detectors along with regular Lidar measurements. The resulting multi-modal detector is simple and effective. Experimental results on the large-scale nuScenes dataset show that our framework improves a strong CenterPoint baseline by a significant 6.6 mAP, and outperforms competing fusion approaches. Code and more visualizations are available at https://tianweiy.github.io/mvp/

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). Builds on 63 of its 73 references that are in the map. First posted on arXiv 12 Nov 2021.

Builds on

63 of 73 references.

References · 63

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2017Mask R-CNNcited by 7,277
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  7. 2015Fast R-CNNcited by 2,933
  8. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  9. 2017Deformable Convolutional Networkscited by 1,488
  10. 2016YOLO9000: Better, Faster, Strongercited by 1,431
  11. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  12. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
38 more
  1. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  2. 2019Objects as Pointscited by 1,175
  3. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  4. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  5. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  6. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  7. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  8. 2017Focal Loss for Dense Object Detectioncited by 752
  9. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  10. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 724
  11. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  12. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  13. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
  14. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  15. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  16. 2017Deep Layer Aggregationcited by 436
  17. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 397
  18. 2018PCN: Point Completion Networkcited by 384
  19. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 368
  20. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  21. 2019Class-balanced Grouping and Sampling for Point Cloud 3D Object Detectioncited by 350
  22. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 327
  23. 2017Sparsity Invariant CNNscited by 326
  24. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 326
  25. 2019STD: Sparse-to-Dense 3D Object Detector for Point Cloudcited by 282
  26. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  27. 2018PU-Net: Point Cloud Upsampling Networkcited by 209
  28. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  29. 20203D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detectioncited by 176
  30. 2019PU-GAN: a Point Cloud Upsampling Adversarial Networkcited by 164
  31. 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
  32. 2019Stereo R-CNN based 3D Object Detection for Autonomous Drivingcited by 163
  33. 2019MVX-Net: Multimodal VoxelNet for 3D Object Detectioncited by 163
  34. 2020RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Drivingcited by 150
  35. 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
  36. 2021PENet: Towards Precise and Efficient Image Guided Depth Completioncited by 146
  37. 2020Pillar-based Object Detection for Autonomous Drivingcited by 137
  38. 2021Probabilistic two-stage detectioncited by 130

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 33; 2023: 51; 2024: 40; 2025: 29; 2026: 13.

166 papers.

Sort:

By area:

  1. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  2. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  3. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
  4. 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
  5. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  6. 2023DDP: Diffusion Model for Dense Visual Predictioncited by 148
  7. 2022Focal Sparse Convolutional Networks for 3D Object Detectioncited by 142
  8. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  9. 2022Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?cited by 128
  10. 2022Vision-Centric BEV Perception: A Surveycited by 126
  11. 2022DeepInteraction: 3D Object Detection via Modality Interactioncited by 125
  12. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
38 more
  1. 2023Collaborative Perception in Autonomous Driving: Methods, Datasets and Challengescited by 87
  2. 2023Virtual Sparse Convolution for Multimodal 3D Object Detectioncited by 81
  3. 2023UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representationcited by 74
  4. 2023FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformercited by 73
  5. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  6. 2022AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detectioncited by 70
  7. 2023MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Drivingcited by 68
  8. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  9. 2022MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detectioncited by 62
  10. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  11. 2023FocalFormer3D : Focusing on Hard Instance for 3D Object Detectioncited by 58
  12. 2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
  13. 2022Fast Point Cloud Generation with Straight Flowscited by 56
  14. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  15. 2022Transformation-Equivariant 3D Object Detection for Autonomous Drivingcited by 52
  16. 2022Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detectioncited by 50
  17. 2022Voxel Field Fusion for 3D Object Detectioncited by 50
  18. 2023VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloudcited by 43
  19. 2023DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge Distillationcited by 41
  20. 2022Towards Long-Tailed 3D Detectioncited by 40
  21. 2022X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentationcited by 38
  22. 2023Understanding the Robustness of 3D Object Detection with Bird's-Eye-View Representations in Autonomous Drivingcited by 37
  23. 2023GraphAlign: Enhancing Accurate Feature Alignment by Graph matching for Multi-Modal 3D Object Detectioncited by 34
  24. 2024GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detectioncited by 33
  25. 2023BEVFusion4D: Learning LiDAR-Camera Fusion Under Bird's-Eye-View via Cross-Modality Guidance and Temporal Aggregationcited by 31
  26. 2023SA-BEV: Generating Semantic-Aware Bird's-Eye-View Feature for Multi-view 3D Object Detectioncited by 30
  27. 2023Fully Sparse Fusion for 3D Object Detectioncited by 29
  28. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  29. 2024Is Your HD Map Constructor Reliable under Sensor Corruptions?cited by 28
  30. 2022Spatial Pruned Sparse Convolution for Efficient 3D Object Detectioncited by 26
  31. 2024Multi-Sensor Object Anomaly Detection: Unifying Appearance, Geometry, and Internal Propertiescited by 26
  32. 2024SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detectioncited by 25
  33. 2023Advancements in Point Cloud Data Augmentation for Deep Learning: A Surveycited by 24
  34. 2024MV2DFusion: Leveraging Modality-Specific Object Semantics for Multi-Modal 3D Detectioncited by 23
  35. 2023OpenSight: A Simple Open-Vocabulary Framework for LiDAR-Based Object Detectioncited by 22
  36. 2023MapPrior: Bird's-Eye View Map Layout Estimation with Generative Modelscited by 21
  37. 2022DetMatch: Two Teachers are Better Than One for Joint 2D and 3D Semi-Supervised Object Detectioncited by 20
  38. 2024Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentationcited by 20

50 most cited of 166 All 166 (JSON)