# ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes

arXiv: https://arxiv.org/abs/2001.10692 · HTML: https://mlsanity.com/paper/2001.10692/

## Key facts

- Cited by 119 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2020.

## Built on by (most cited first, 30 of 119)

- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022)
- [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://mlsanity.com/paper/2106.01178/) (2021)
- [EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI](https://mlsanity.com/paper/2312.16170/) (2023)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [LiDAR R-CNN: An Efficient and Universal 3D Object Detector](https://mlsanity.com/paper/2103.15297/) (2021)
- [DeepInteraction: 3D Object Detection via Modality Interaction](https://mlsanity.com/paper/2208.11112/) (2022)
- [SAT: 2D Semantics Assisted Training for 3D Visual Grounding](https://mlsanity.com/paper/2105.11450/) (2021)
- [FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection](https://mlsanity.com/paper/2112.00322/) (2021)
- [Surface Representation for Point Clouds](https://mlsanity.com/paper/2205.05740/) (2022)
- [X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning](https://mlsanity.com/paper/2203.00843/) (2022)
- [UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding](https://mlsanity.com/paper/2212.00836/) (2022)
- [Pri3D: Can 3D Priors Help 2D Representation Learning?](https://mlsanity.com/paper/2104.11225/) (2021)
- [Object DGCNN: 3D Object Detection using Dynamic Graphs](https://mlsanity.com/paper/2110.06923/) (2021)
- [CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds](https://mlsanity.com/paper/2210.04264/) (2022)
- [4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding](https://mlsanity.com/paper/2112.02990/) (2021)
- [NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2307.14620/) (2023)
- [Voxel Field Fusion for 3D Object Detection](https://mlsanity.com/paper/2205.15938/) (2022)
- [EPNet++: Cascade Bi-directional Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2112.11088/) (2021)
- [NeRF-RPN: A general framework for object detection in NeRFs](https://mlsanity.com/paper/2211.11646/) (2022)
- [VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud](https://mlsanity.com/paper/2303.14408/) (2023)
- [Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding](https://mlsanity.com/paper/2211.14241/) (2022)
- [RBGNet: Ray-based Grouping for 3D Object Detection](https://mlsanity.com/paper/2204.02251/) (2022)
- [Cubify Anything: Scaling Indoor 3D Object Detection](https://mlsanity.com/paper/2412.04458/) (2024)
- [STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenes](https://mlsanity.com/paper/2204.01026/) (2022)
- [Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining](https://mlsanity.com/paper/2104.04687/) (2021)

## Abstract (arXiv)

3D object detection has seen quick progress thanks to advances in deep learning on point clouds. A few recent works have even shown state-of-the-art performance with just point clouds input (e.g. VoteNet). However, point cloud data have inherent limitations. They are sparse, lack color information and often suffer from sensor noise. Images, on the other hand, have high resolution and rich texture. Thus they can complement the 3D geometry provided by point clouds. Yet how to effectively use image information to assist point cloud based detection is still an open question. In this work, we build on top of VoteNet and propose a 3D detection architecture called ImVoteNet specialized for RGB-D scenes. ImVoteNet is based on fusing 2D votes in images and 3D votes in point clouds. Compared to prior work on multi-modal detection, we explicitly extract both geometric and semantic features from the 2D images. We leverage camera parameters to lift these features to 3D. To improve the synergy of 2D-3D feature fusion, we also propose a multi-tower training scheme. We validate our model on the challenging SUN RGB-D dataset, advancing state-of-the-art results by 5.7 mAP. We also provide rich ablation studies to analyze the contribution of each design choice.