# Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation

arXiv: https://arxiv.org/abs/2403.06401 · HTML: https://mlsanity.com/paper/2403.06401/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 68 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [MEMO: Test Time Robustness via Adaptation and Augmentation](https://mlsanity.com/paper/2110.09506/) (2021)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [Parameter-free Online Test-time Adaptation](https://mlsanity.com/paper/2201.05718/) (2022)
- [Stratified Transformer for 3D Point Cloud Segmentation](https://mlsanity.com/paper/2203.14508/) (2022)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation](https://mlsanity.com/paper/2001.10331/) (2020)
- [Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning](https://mlsanity.com/paper/1710.04043/) (2017)
- [FocalClick: Towards Practical Interactive Image Segmentation](https://mlsanity.com/paper/2204.02574/) (2022)
- [Iteratively Trained Interactive Segmentation](https://mlsanity.com/paper/1805.04398/) (2018)
- [PseudoClick: Interactive Image Segmentation with Click Imitation](https://mlsanity.com/paper/2207.05282/) (2022)
- [EdgeFlow: Achieving Practical Interactive Segmentation with Edge-Guided Flow](https://mlsanity.com/paper/2109.09406/) (2021)
- [Towards Open-Set Test-Time Adaptation Utilizing the Wisdom of Crowds in Entropy Minimization](https://mlsanity.com/paper/2308.06879/) (2023)
- [Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections](https://mlsanity.com/paper/1911.12709/) (2019)
- [GrowSP: Unsupervised Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2305.16404/) (2023)
- [Less is More: Reducing Task and Model Complexity for 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2303.11203/) (2023)
- [AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation](https://mlsanity.com/paper/2306.00977/) (2023)
- [Interactive Object Segmentation in 3D Point Clouds](https://mlsanity.com/paper/2204.07183/) (2022)
- [InterFormer: Real-time Interactive Image Segmentation](https://mlsanity.com/paper/2304.02942/) (2023)
- [Focused and Collaborative Feedback Integration for Interactive Image Segmentation](https://mlsanity.com/paper/2303.11880/) (2023)

## Built on by (most cited first, 3 of 3)

- [Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes](https://mlsanity.com/paper/2505.01726/) (2025)
- [PinPoint3D: Fine-Grained 3D Part Segmentation from a Few Clicks](https://mlsanity.com/paper/2509.25970/) (2025)
- [ClickSeg3D: Few-Click Interactive Segmentation via Semantic Embeddings](https://mlsanity.com/paper/2605.08925/) (2026)

## Abstract (arXiv)

Existing interactive point cloud segmentation approaches primarily focus on the object segmentation, which aim to determine which points belong to the object of interest guided by user interactions. This paper concentrates on an unexplored yet meaningful task, i.e., interactive point cloud semantic segmentation, which assigns high-quality semantic labels to all points in a scene with user corrective clicks. Concretely, we presents the first interactive framework for point cloud semantic segmentation, named InterPCSeg, which seamlessly integrates with off-the-shelf semantic segmentation networks without offline re-training, enabling it to run in an on-the-fly manner. To achieve online refinement, we treat user interactions as sparse training examples during the test-time. To address the instability caused by the sparse supervision, we design a stabilization energy to regulate the test-time training process. For objective and reproducible evaluation, we develop an interaction simulation scheme tailored for the interactive point cloud semantic segmentation task. We evaluate our framework on the S3DIS and ScanNet datasets with off-the-shelf segmentation networks, incorporating interactions from both the proposed interaction simulator and real users. Quantitative and qualitative experimental results demonstrate the efficacy of our framework in refining the semantic segmentation results with user interactions. The source code will be publicly available.