# Multi-view PointNet for 3D Scene Understanding

arXiv: https://arxiv.org/abs/1909.13603 · HTML: https://mlsanity.com/paper/1909.13603/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2019.

## Built on by (most cited first, 30 of 76)

- [PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection](https://mlsanity.com/paper/2102.00463/) (2021)
- [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://mlsanity.com/paper/2106.01178/) (2021)
- [SAM3D: Segment Anything in 3D Scenes](https://mlsanity.com/paper/2306.03908/) (2023)
- [NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes](https://mlsanity.com/paper/2111.13260/) (2021)
- [Bidirectional Projection Network for Cross Dimension Scene Understanding](https://mlsanity.com/paper/2103.14326/) (2021)
- [NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space](https://mlsanity.com/paper/2309.14616/) (2023)
- [X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning](https://mlsanity.com/paper/2203.00843/) (2022)
- [SQN: Weakly-Supervised Semantic Segmentation of Large-Scale 3D Point Clouds](https://mlsanity.com/paper/2104.04891/) (2021)
- [Grounding 3D Object Affordance from 2D Interactions in Images](https://mlsanity.com/paper/2303.10437/) (2023)
- [Learning 3D Semantic Segmentation with only 2D Image Supervision](https://mlsanity.com/paper/2110.11325/) (2021)
- [Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation](https://mlsanity.com/paper/2204.07548/) (2022)
- [Sparse-to-dense Feature Matching: Intra and Inter domain Cross-modal Learning in Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2107.14724/) (2021)
- [Visual Reinforcement Learning with Self-Supervised 3D Representations](https://mlsanity.com/paper/2210.07241/) (2022)
- [Self-positioning Point-based Transformer for Point Cloud Understanding](https://mlsanity.com/paper/2303.16450/) (2023)
- [Instance Neural Radiance Field](https://mlsanity.com/paper/2304.04395/) (2023)
- [Point Cloud Augmentation with Weighted Local Transformations](https://mlsanity.com/paper/2110.05379/) (2021)
- [PartSLIP++: Enhancing Low-Shot 3D Part Segmentation via Multi-View Instance Segmentation and Maximum Likelihood Estimation](https://mlsanity.com/paper/2312.03015/) (2023)
- [UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenes](https://mlsanity.com/paper/2303.14541/) (2023)
- [DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentation](https://mlsanity.com/paper/2503.18944/) (2025)
- [Deep Learning Based 3D Segmentation: A Survey](https://mlsanity.com/paper/2103.05423/) (2021)
- [ODIN: A Single Model for 2D and 3D Segmentation](https://mlsanity.com/paper/2401.02416/) (2024)
- [Voint Cloud: Multi-View Point Cloud Representation for 3D Understanding](https://mlsanity.com/paper/2111.15363/) (2021)
- [OV-NeRF: Open-vocabulary Neural Radiance Fields with Vision and Language Foundation Models for 3D Semantic Understanding](https://mlsanity.com/paper/2402.04648/) (2024)
- [Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding](https://mlsanity.com/paper/2408.13024/) (2024)
- [GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding](https://mlsanity.com/paper/2311.11863/) (2023)
- [Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training](https://mlsanity.com/paper/2311.01734/) (2023)
- [LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understanding](https://mlsanity.com/paper/2412.17635/) (2024)
- [2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision](https://mlsanity.com/paper/2310.12817/) (2023)
- [Dual Adaptive Transformations for Weakly Supervised Point Cloud Segmentation](https://mlsanity.com/paper/2207.09084/) (2022)
- [A Unified Framework for 3D Scene Understanding](https://mlsanity.com/paper/2407.03263/) (2024)

## Abstract (arXiv)

Fusion of 2D images and 3D point clouds is important because information from dense images can enhance sparse point clouds. However, fusion is challenging because 2D and 3D data live in different spaces. In this work, we propose MVPNet (Multi-View PointNet), where we aggregate 2D multi-view image features into 3D point clouds, and then use a point based network to fuse the features in 3D canonical space to predict 3D semantic labels. To this end, we introduce view selection along with a 2D-3D feature aggregation module. Extensive experiments show the benefit of leveraging features from dense images and reveal superior robustness to varying point cloud density compared to 3D-only methods. On the ScanNetV2 benchmark, our MVPNet significantly outperforms prior point cloud based approaches on the task of 3D Semantic Segmentation. It is much faster to train than the large networks of the sparse voxel approach. We provide solid ablation studies to ease the future design of 2D-3D fusion methods and their extension to other tasks, as we showcase for 3D instance segmentation.