# Virtual Multi-view Fusion for 3D Semantic Segmentation

arXiv: https://arxiv.org/abs/2007.13138 · HTML: https://mlsanity.com/paper/2007.13138/

## Key facts

- Cited by 99 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2020.

## Built on by (most cited first, 30 of 99)

- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [SAM3D: Segment Anything in 3D Scenes](https://mlsanity.com/paper/2306.03908/) (2023)
- [Mix3D: Out-of-Context Data Augmentation for 3D Scenes](https://mlsanity.com/paper/2110.02210/) (2021)
- [Neural 3D Scene Reconstruction with the Manhattan-world Assumption](https://mlsanity.com/paper/2205.02836/) (2022)
- [NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes](https://mlsanity.com/paper/2111.13260/) (2021)
- [LENS: Localization enhanced by NeRF synthesis](https://mlsanity.com/paper/2110.06558/) (2021)
- [Fast Point Transformer](https://mlsanity.com/paper/2112.04702/) (2021)
- [Bidirectional Projection Network for Cross Dimension Scene Understanding](https://mlsanity.com/paper/2103.14326/) (2021)
- [SAI3D: Segment Any Instance in 3D Scenes](https://mlsanity.com/paper/2312.11557/) (2023)
- [One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation](https://mlsanity.com/paper/2104.02246/) (2021)
- [Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2110.08188/) (2021)
- [OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation](https://mlsanity.com/paper/2309.00616/) (2023)
- [SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2204.03636/) (2022)
- [SQN: Weakly-Supervised Semantic Segmentation of Large-Scale 3D Point Clouds](https://mlsanity.com/paper/2104.04891/) (2021)
- [Learning 3D Semantic Segmentation with only 2D Image Supervision](https://mlsanity.com/paper/2110.11325/) (2021)
- [Contrastive Boundary Learning for Point Cloud Segmentation](https://mlsanity.com/paper/2203.05272/) (2022)
- [4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding](https://mlsanity.com/paper/2112.02990/) (2021)
- [3D-to-2D Distillation for Indoor Scene Parsing](https://mlsanity.com/paper/2104.02243/) (2021)
- [Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation](https://mlsanity.com/paper/2204.07548/) (2022)
- [Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervision](https://mlsanity.com/paper/2303.03361/) (2023)
- [SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2311.17707/) (2023)
- [SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data](https://mlsanity.com/paper/2105.08612/) (2021)
- [RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection](https://mlsanity.com/paper/2108.07794/) (2021)
- [GrowSP: Unsupervised Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2305.16404/) (2023)
- [Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining](https://mlsanity.com/paper/2104.04687/) (2021)
- [BuildingNet: Learning to Label 3D Buildings](https://mlsanity.com/paper/2110.04955/) (2021)
- [DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentation](https://mlsanity.com/paper/2503.18944/) (2025)
- [PointConvFormer: Revenge of the Point-based Convolution](https://mlsanity.com/paper/2208.02879/) (2022)
- [S4C: Self-Supervised Semantic Scene Completion with Neural Fields](https://mlsanity.com/paper/2310.07522/) (2023)
- [Divide and Conquer: 3D Point Cloud Instance Segmentation With Point-Wise Binarization](https://mlsanity.com/paper/2207.11209/) (2022)

## Abstract (arXiv)

Semantic segmentation of 3D meshes is an important problem for 3D scene understanding. In this paper we revisit the classic multiview representation of 3D meshes and study several techniques that make them effective for 3D semantic segmentation of meshes. Given a 3D mesh reconstructed from RGBD sensors, our method effectively chooses different virtual views of the 3D mesh and renders multiple 2D channels for training an effective 2D semantic segmentation model. Features from multiple per view predictions are finally fused on 3D mesh vertices to predict mesh semantic segmentation labels. Using the large scale indoor 3D semantic segmentation benchmark of ScanNet, we show that our virtual views enable more effective training of 2D semantic segmentation networks than previous multiview approaches. When the 2D per pixel predictions are aggregated on 3D surfaces, our virtual multiview fusion method is able to achieve significantly better 3D semantic segmentation results compared to all prior multiview approaches and competitive with recent 3D convolution approaches.