# Interactive Object Segmentation in 3D Point Clouds

arXiv: https://arxiv.org/abs/2204.07183 · HTML: https://mlsanity.com/paper/2204.07183/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 29 of 29)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts](https://mlsanity.com/paper/2012.09165/) (2020)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/1906.01140/) (2019)
- [3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2003.13867/) (2020)
- [Hierarchical Aggregation for 3D Instance Segmentation](https://mlsanity.com/paper/2108.02350/) (2021)
- [Large-scale interactive object segmentation with human annotators](https://mlsanity.com/paper/1903.10830/) (2019)
- [Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks](https://mlsanity.com/paper/2108.07478/) (2021)
- [One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation](https://mlsanity.com/paper/2104.02246/) (2021)
- [Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2110.08188/) (2021)
- [Associatively Segmenting Instances and Semantics in Point Clouds](https://mlsanity.com/paper/1902.09852/) (2019)
- [Iteratively Trained Interactive Segmentation](https://mlsanity.com/paper/1805.04398/) (2018)
- [Weakly Supervised Semantic Point Cloud Segmentation:Towards 10X Fewer Labels](https://mlsanity.com/paper/2004.04091/) (2020)
- [Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections](https://mlsanity.com/paper/1911.12709/) (2019)
- [ILabel: Interactive Neural Scene Labelling](https://mlsanity.com/paper/2111.14637/) (2021)
- [Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point Clouds of Wild Scenes](https://mlsanity.com/paper/2004.12498/) (2020)
- [Interactive Annotation of 3D Object Geometry using 2D Scribbles](https://mlsanity.com/paper/2008.10719/) (2020)
- [SemanticPaint: A Framework for the Interactive Segmentation of 3D Scenes](https://mlsanity.com/paper/1510.03727/) (2015)

## Built on by (most cited first, 23 of 23)

- [Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels](https://mlsanity.com/paper/2312.17232/) (2023)
- [Point-SAM: Promptable 3D Segmentation Model for Point Clouds](https://mlsanity.com/paper/2406.17741/) (2024)
- [AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation](https://mlsanity.com/paper/2306.00977/) (2023)
- [3D Segmentation of Humans in Point Clouds with Synthetic Data](https://mlsanity.com/paper/2212.00786/) (2022)
- [A Unified Framework for 3D Scene Understanding](https://mlsanity.com/paper/2407.03263/) (2024)
- [OpenAnnotate3D: Open-Vocabulary Auto-Labeling System for Multi-modal 3D Data](https://mlsanity.com/paper/2310.13398/) (2023)
- [LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectories](https://mlsanity.com/paper/2311.12174/) (2023)
- [Neural Interactive Keypoint Detection](https://mlsanity.com/paper/2308.10174/) (2023)
- [iSegMan: Interactive Segment-and-Manipulate 3D Gaussians](https://mlsanity.com/paper/2505.11934/) (2025)
- [FIction: 4D Future Interaction Prediction from Video](https://mlsanity.com/paper/2412.00932/) (2024)
- [WildSeg3D: Segment Any 3D Objects in the Wild from 2D Images](https://mlsanity.com/paper/2503.08407/) (2025)
- [Interactive4D: Interactive 4D LiDAR Segmentation](https://mlsanity.com/paper/2410.08206/) (2024)
- [SALT: A Flexible Semi-Automatic Labeling Tool for General LiDAR Point Clouds with Cross-Scene Adaptability and 4D Consistency](https://mlsanity.com/paper/2503.23980/) (2025)
- [Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2403.06401/) (2024)
- [AutoInst: Automatic Instance-Based Segmentation of LiDAR 3D Scans](https://mlsanity.com/paper/2403.16318/) (2024)
- [Scale Disparity of Instances in Interactive Point Cloud Segmentation](https://mlsanity.com/paper/2407.14009/) (2024)
- [Easy3D: A Simple Yet Effective Method for 3D Interactive Segmentation](https://mlsanity.com/paper/2504.11024/) (2025)
- [Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes](https://mlsanity.com/paper/2505.01726/) (2025)
- [SUM Parts: Benchmarking Part-Level Semantic Segmentation of Urban Meshes](https://mlsanity.com/paper/2503.15300/) (2025)
- [PinPoint3D: Fine-Grained 3D Part Segmentation from a Few Clicks](https://mlsanity.com/paper/2509.25970/) (2025)
- [Viewpoint Recommendation for Point Cloud Labeling through Interaction Cost Modeling](https://mlsanity.com/paper/2602.10871/) (2026)
- [ClickSeg3D: Few-Click Interactive Segmentation via Semantic Embeddings](https://mlsanity.com/paper/2605.08925/) (2026)
- [NegROI: Click-Centric Uncertainty-Guided Refinement with Scene-Conditioned Negative Prompts for Robust Interactive 3D Segmentation](https://mlsanity.com/paper/2607.05955/) (2026)

## Abstract (arXiv)

We propose an interactive approach for 3D instance segmentation, where users can iteratively collaborate with a deep learning model to segment objects in a 3D point cloud directly. Current methods for 3D instance segmentation are generally trained in a fully-supervised fashion, which requires large amounts of costly training labels, and does not generalize well to classes unseen during training. Few works have attempted to obtain 3D segmentation masks using human interactions. Existing methods rely on user feedback in the 2D image domain. As a consequence, users are required to constantly switch between 2D images and 3D representations, and custom architectures are employed to combine multiple input modalities. Therefore, integration with existing standard 3D models is not straightforward. The core idea of this work is to enable users to interact directly with 3D point clouds by clicking on desired 3D objects of interest~(or their background) to interactively segment the scene in an open-world setting. Specifically, our method does not require training data from any target domain, and can adapt to new environments where no appropriate training sets are available. Our system continuously adjusts the object segmentation based on the user feedback and achieves accurate dense 3D segmentation masks with minimal human effort (few clicks per object). Besides its potential for efficient labeling of large-scale and varied 3D datasets, our approach, where the user directly interacts with the 3D environment, enables new applications in AR/VR and human-robot interaction.