# Virtual Sparse Convolution for Multimodal 3D Object Detection

arXiv: https://arxiv.org/abs/2303.02314 · HTML: https://mlsanity.com/paper/2303.02314/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 51 references that are in the map.
- First posted on arXiv 4 Mar 2023.

## Builds on (most cited first, 30 of 41)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [Multi-Task Multi-Sensor Fusion for 3D Object Detection](https://mlsanity.com/paper/2012.12397/) (2020)
- [FUTR3D: A Unified Sensor Fusion Framework for 3D Detection](https://mlsanity.com/paper/2203.10642/) (2022)
- [3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection](https://mlsanity.com/paper/2004.12636/) (2020)
- [Multimodal Virtual Point 3D Detection](https://mlsanity.com/paper/2111.06881/) (2021)
- [MVX-Net: Multimodal VoxelNet for 3D Object Detection](https://mlsanity.com/paper/1904.01649/) (2019)
- [ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection](https://mlsanity.com/paper/2103.05346/) (2021)
- [PENet: Towards Precise and Efficient Image Guided Depth Completion](https://mlsanity.com/paper/2103.00783/) (2021)
- [Focal Sparse Convolutional Networks for 3D Object Detection](https://mlsanity.com/paper/2204.12463/) (2022)
- [IoU Loss for 2D/3D Object Detection](https://mlsanity.com/paper/1908.03851/) (2019)
- [SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud](https://mlsanity.com/paper/2104.09804/) (2021)
- [Improving 3D Object Detection with Channel-wise Transformer](https://mlsanity.com/paper/2108.10723/) (2021)
- [SESS: Self-Ensembling Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/1912.11803/) (2019)
- [3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/2012.04355/) (2020)

## Built on by (most cited first, 30 of 81)

- [UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representation](https://mlsanity.com/paper/2308.07732/) (2023)
- [Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era](https://mlsanity.com/paper/2305.06131/) (2023)
- [IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection](https://mlsanity.com/paper/2403.15241/) (2024)
- [Detecting As Labeling: Rethinking LiDAR-camera Fusion in 3D Object Detection](https://mlsanity.com/paper/2311.07152/) (2023)
- [OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagation](https://mlsanity.com/paper/2403.19580/) (2024)
- [V2X-R: Cooperative LiDAR-4D Radar Fusion with Denoising Diffusion for 3D Object Detection](https://mlsanity.com/paper/2411.08402/) (2024)
- [MCTrack: A Unified 3D Multi-Object Tracking Framework for Autonomous Driving](https://mlsanity.com/paper/2409.16149/) (2024)
- [VoxelNextFusion: A Simple, Unified and Effective Voxel Fusion Framework for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2401.02702/) (2024)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Commonsense Prototype for Outdoor Unsupervised 3D Object Detection](https://mlsanity.com/paper/2404.16493/) (2024)
- [DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation](https://mlsanity.com/paper/2503.11122/) (2025)
- [Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection](https://mlsanity.com/paper/2402.18493/) (2024)
- [MonoTAKD: Teaching Assistant Knowledge Distillation for Monocular 3D Object Detection](https://mlsanity.com/paper/2404.04910/) (2024)
- [PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via Foundation Models](https://mlsanity.com/paper/2403.06403/) (2024)
- [RobMOT: Robust 3D Multi-Object Tracking by Observational Noise and State Estimation Drift Mitigation on LiDAR PointCloud](https://mlsanity.com/paper/2405.11536/) (2024)
- [PeP: a Point enhanced Painting method for unified point cloud tasks](https://mlsanity.com/paper/2310.07591/) (2023)
- [Density-guided Translator Boosts Synthetic-to-Real Unsupervised Domain Adaptive Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2403.18469/) (2024)
- [Learning to Detect Objects from Multi-Agent LiDAR Scans without Manual Labels](https://mlsanity.com/paper/2503.08421/) (2025)
- [BiTrack: Bidirectional Offline 3D Multi-Object Tracking Using Camera-LiDAR Data](https://mlsanity.com/paper/2406.18414/) (2024)
- [FSF-Net: Enhance 4D Occupancy Forecasting with Coarse BEV Scene Flow for Autonomous Driving](https://mlsanity.com/paper/2409.15841/) (2024)
- [SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts](https://mlsanity.com/paper/2503.06467/) (2025)
- [Multi-Modal 3D Object Detection by Box Matching](https://mlsanity.com/paper/2305.07713/) (2023)
- [Improving Robustness of LiDAR-Camera Fusion Model against Weather Corruption from Fusion Strategy Perspective](https://mlsanity.com/paper/2402.02738/) (2024)
- [RePLAy: Remove Projective LiDAR Depthmap Artifacts via Exploiting Epipolar Geometry](https://mlsanity.com/paper/2407.19154/) (2024)
- [GATE3D: Generalized Attention-based Task-synergized Estimation in 3D*](https://mlsanity.com/paper/2504.11014/) (2025)
- [GraphRelate3D: Context-Dependent 3D Object Detection with Inter-Object Relationship Graphs](https://mlsanity.com/paper/2405.06782/) (2024)
- [LEROjD: Lidar Extended Radar-Only Object Detection](https://mlsanity.com/paper/2409.05564/) (2024)
- [PhantomLiDAR: Cross-modality Signal Injection Attacks against LiDAR](https://mlsanity.com/paper/2409.17907/) (2024)
- [HybridTrack: A Hybrid Approach for Robust Multi-Object Tracking](https://mlsanity.com/paper/2501.01275/) (2025)
- [Rethinking Backbone Design for Lightweight 3D Object Detection in LiDAR](https://mlsanity.com/paper/2508.00744/) (2025)

## Abstract (arXiv)

Recently, virtual/pseudo-point-based 3D object detection that seamlessly fuses RGB images and LiDAR data by depth completion has gained great attention. However, virtual points generated from an image are very dense, introducing a huge amount of redundant computation during detection. Meanwhile, noises brought by inaccurate depth completion significantly degrade detection precision. This paper proposes a fast yet effective backbone, termed VirConvNet, based on a new operator VirConv (Virtual Sparse Convolution), for virtual-point-based 3D object detection. VirConv consists of two key designs: (1) StVD (Stochastic Voxel Discard) and (2) NRConv (Noise-Resistant Submanifold Convolution). StVD alleviates the computation problem by discarding large amounts of nearby redundant voxels. NRConv tackles the noise problem by encoding voxel features in both 2D image and 3D LiDAR space. By integrating VirConv, we first develop an efficient pipeline VirConv-L based on an early fusion design. Then, we build a high-precision pipeline VirConv-T based on a transformed refinement scheme. Finally, we develop a semi-supervised pipeline VirConv-S based on a pseudo-label framework. On the KITTI car 3D detection test leaderboard, our VirConv-L achieves 85% AP with a fast running speed of 56ms. Our VirConv-T and VirConv-S attains a high-precision of 86.3% and 87.2% AP, and currently rank 2nd and 1st, respectively. The code is available at https://github.com/hailanyi/VirConv.