# Open-world Point Cloud Semantic Segmentation: A Human-in-the-loop Framework

arXiv: https://arxiv.org/abs/2508.04962 · HTML: https://mlsanity.com/paper/2508.04962/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 30 of 34)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [Stratified Transformer for 3D Point Cloud Segmentation](https://mlsanity.com/paper/2203.14508/) (2022)
- [OctFormer: Octree-based Transformers for 3D Point Clouds](https://mlsanity.com/paper/2305.03045/) (2023)
- [Fast Point Transformer](https://mlsanity.com/paper/2112.04702/) (2021)
- [OneFormer3D: One Transformer for Unified Point Cloud Segmentation](https://mlsanity.com/paper/2311.14405/) (2023)
- [Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02909/) (2022)
- [Deep Projective 3D Semantic Segmentation](https://mlsanity.com/paper/1705.03428/) (2017)
- [Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2006.12052/) (2020)
- [Generalized Few-shot Semantic Segmentation](https://mlsanity.com/paper/2010.05210/) (2020)
- [OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation](https://mlsanity.com/paper/2403.14418/) (2024)
- [Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning](https://mlsanity.com/paper/1906.02944/) (2019)
- [Open-world Semantic Segmentation for LIDAR Point Clouds](https://mlsanity.com/paper/2207.01452/) (2022)
- [Prototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2305.14335/) (2023)
- [Prototype-based Incremental Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2012.01415/) (2020)
- [Boosting Few-shot 3D Point Cloud Segmentation via Query-Guided Enhancement](https://mlsanity.com/paper/2308.03177/) (2023)
- [Rethinking Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2403.00592/) (2024)
- [No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation](https://mlsanity.com/paper/2404.04050/) (2024)
- [Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2403.07630/) (2024)
- [Weakly Supervised 3D Point Cloud Segmentation via Multi-Prototype Learning](https://mlsanity.com/paper/2205.03137/) (2022)

## Abstract (arXiv)

Open-world point cloud semantic segmentation (OW-Seg) aims to predict point labels of both base and novel classes in real-world scenarios. However, existing methods rely on resource-intensive offline incremental learning or densely annotated support data, limiting their practicality. To address these limitations, we propose HOW-Seg, the first human-in-the-loop framework for OW-Seg. Specifically, we construct class prototypes, the fundamental segmentation units, directly on the query data, avoiding the prototype bias caused by intra-class distribution shifts between the support and query data. By leveraging sparse human annotations as guidance, HOW-Seg enables prototype-based segmentation for both base and novel classes. Considering the lack of granularity of initial prototypes, we introduce a hierarchical prototype disambiguation mechanism to refine ambiguous prototypes, which correspond to annotations of different classes. To further enrich contextual awareness, we employ a dense conditional random field (CRF) upon the refined prototypes to optimize their label assignments. Through iterative human feedback, HOW-Seg dynamically improves its predictions, achieving high-quality segmentation for both base and novel classes. Experiments demonstrate that with sparse annotations (e.g., one-novel-class-one-click), HOW-Seg matches or surpasses the state-of-the-art generalized few-shot segmentation (GFS-Seg) method under the 5-shot setting. When using advanced backbones (e.g., Stratified Transformer) and denser annotations (e.g., 10 clicks per sub-scene), HOW-Seg achieves 85.27% mIoU on S3DIS and 66.37% mIoU on ScanNetv2, significantly outperforming alternatives.