# SemanticPaint: A Framework for the Interactive Segmentation of 3D Scenes

arXiv: https://arxiv.org/abs/1510.03727 · HTML: https://mlsanity.com/paper/1510.03727/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2015.

## Built on by (most cited first, 8 of 8)

- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understanding](https://mlsanity.com/paper/2210.03043/) (2022)
- [AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation](https://mlsanity.com/paper/2306.00977/) (2023)
- [ILabel: Interactive Neural Scene Labelling](https://mlsanity.com/paper/2111.14637/) (2021)
- [Interactive Object Segmentation in 3D Point Clouds](https://mlsanity.com/paper/2204.07183/) (2022)
- [VIBUS: Data-efficient 3D Scene Parsing with VIewpoint Bottleneck and Uncertainty-Spectrum Modeling](https://mlsanity.com/paper/2210.11472/) (2022)
- [Neural Interactive Keypoint Detection](https://mlsanity.com/paper/2308.10174/) (2023)
- [Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2403.06401/) (2024)

## Abstract (arXiv)

We present an open-source, real-time implementation of SemanticPaint, a system for geometric reconstruction, object-class segmentation and learning of 3D scenes. Using our system, a user can walk into a room wearing a depth camera and a virtual reality headset, and both densely reconstruct the 3D scene and interactively segment the environment into object classes such as 'chair', 'floor' and 'table'. The user interacts physically with the real-world scene, touching objects and using voice commands to assign them appropriate labels. These user-generated labels are leveraged by an online random forest-based machine learning algorithm, which is used to predict labels for previously unseen parts of the scene. The entire pipeline runs in real time, and the user stays 'in the loop' throughout the process, receiving immediate feedback about the progress of the labelling and interacting with the scene as necessary to refine the predicted segmentation.