mlsanityData to 14 Oct 2026

Multi-view PointNet for 3D Scene Understanding

arXiv 1909.13603 ↗First posted 30 Sep 2019cs.CVcited by 76

Not on the map. Why?

Abstract (arXiv)

Fusion of 2D images and 3D point clouds is important because information from dense images can enhance sparse point clouds. However, fusion is challenging because 2D and 3D data live in different spaces. In this work, we propose MVPNet (Multi-View PointNet), where we aggregate 2D multi-view image features into 3D point clouds, and then use a point based network to fuse the features in 3D canonical space to predict 3D semantic labels. To this end, we introduce view selection along with a 2D-3D feature aggregation module. Extensive experiments show the benefit of leveraging features from dense images and reveal superior robustness to varying point cloud density compared to 3D-only methods. On the ScanNetV2 benchmark, our MVPNet significantly outperforms prior point cloud based approaches on the task of 3D Semantic Segmentation. It is much faster to train than the large networks of the sparse voxel approach. We provide solid ablation studies to ease the future design of 2D-3D fusion methods and their extension to other tasks, as we showcase for 3D instance segmentation.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 12; 2023: 23; 2024: 15; 2025: 7; 2026: 6.

76 papers.

Sort:

By area:

  1. 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
  2. 2021ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detectioncited by 157
  3. 2023SAM3D: Segment Anything in 3D Scenescited by 157
  4. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  5. 2021Bidirectional Projection Network for Cross Dimension Scene Understandingcited by 95
  6. 2023NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Spacecited by 90
  7. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  8. 2021SQN: Weakly-Supervised Semantic Segmentation of Large-Scale 3D Point Cloudscited by 64
  9. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  10. 2021Learning 3D Semantic Segmentation with only 2D Image Supervisioncited by 60
  11. 2022Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentationcited by 47
  12. 2021Sparse-to-dense Feature Matching: Intra and Inter domain Cross-modal Learning in Domain Adaptation for 3D Semantic Segmentationcited by 46
38 more
  1. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  2. 2023Self-positioning Point-based Transformer for Point Cloud Understandingcited by 40
  3. 2023Instance Neural Radiance Fieldcited by 40
  4. 2021Point Cloud Augmentation with Weighted Local Transformationscited by 39
  5. 2023PartSLIP++: Enhancing Low-Shot 3D Part Segmentation via Multi-View Instance Segmentation and Maximum Likelihood Estimationcited by 37
  6. 2023UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenescited by 31
  7. 2025DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentationcited by 31
  8. 2021Deep Learning Based 3D Segmentation: A Surveycited by 30
  9. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  10. 2021Voint Cloud: Multi-View Point Cloud Representation for 3D Understandingcited by 22
  11. 2024OV-NeRF: Open-vocabulary Neural Radiance Fields with Vision and Language Foundation Models for 3D Semantic Understandingcited by 22
  12. 2024Learning 2D Invariant Affordance Knowledge for 3D Affordance Groundingcited by 22
  13. 2023GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understandingcited by 20
  14. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 18
  15. 2024LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understandingcited by 17
  16. 20232D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervisioncited by 16
  17. 2022Dual Adaptive Transformations for Weakly Supervised Point Cloud Segmentationcited by 15
  18. 2024A Unified Framework for 3D Scene Understandingcited by 15
  19. 2023PointMBF: A Multi-scale Bidirectional Fusion Network for Unsupervised RGB-D Point Cloud Registrationcited by 14
  20. 2025V2-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondencecited by 14
  21. 2022Learning Inter-Superpoint Affinity for Weakly Supervised 3D Instance Segmentationcited by 13
  22. 20243D Weakly Supervised Semantic Segmentation with 2D Vision-Language Guidancecited by 12
  23. 2022SemAffiNet: Semantic-Affine Transformation for Point Cloud Segmentationcited by 11
  24. 20243x2: 3D Object Part Segmentation by 2D Semantic Correspondencescited by 10
  25. 2023A Review of Panoptic Segmentation for Mobile Mapping Point Cloudscited by 8
  26. 2022MVTN: Learning Multi-View Transformations for 3D Understandingcited by 7
  27. 2023Leveraging Large-Scale Pretrained Vision Foundation Models for Label-Efficient 3D Point Cloud Segmentationcited by 7
  28. 2025Pointmap Association and Piecewise-Plane Constraint for Consistent and Compact 3D Gaussian Segmentation Fieldcited by 7
  29. 2023Continual Road-Scene Semantic Segmentation via Feature-Aligned Symmetric Multi-Modal Networkcited by 6
  30. 2022AziNorm: Exploiting the Radial Symmetry of Point Cloud for Azimuth-Normalized 3D Perceptioncited by 5
  31. 2023Full Point Encoding for Local Feature Aggregation in 3D Point Cloudscited by 5
  32. 2023Multi-task View Synthesis with Neural Radiance Fieldscited by 5
  33. 2024Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Imagescited by 5
  34. 2022Collaborative Propagation on Multiple Instance Graphs for 3D Instance Segmentation with Single-point Supervisioncited by 4
  35. 2024Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictionscited by 4
  36. 2021Similarity-Aware Fusion Network for 3D Semantic Segmentationcited by 3
  37. 2022Towards Deeper and Better Multi-view Feature Fusion for 3D Semantic Segmentationcited by 3
  38. 2023Variational Relational Point Completion Network for Robust 3D Classificationcited by 3

50 most cited of 76 All 76 (JSON)