# ILabel: Interactive Neural Scene Labelling

arXiv: https://arxiv.org/abs/2111.14637 · HTML: https://mlsanity.com/paper/2111.14637/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 28 of 28)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM](https://mlsanity.com/paper/1804.00874/) (2018)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)
- [DeepFactors: Real-Time Probabilistic Dense Monocular SLAM](https://mlsanity.com/paper/2001.05049/) (2020)
- [BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration](https://mlsanity.com/paper/1604.01093/) (2016)
- [Fusion++: Volumetric Object-Level SLAM](https://mlsanity.com/paper/1808.08378/) (2018)
- [PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things](https://mlsanity.com/paper/1903.01177/) (2019)
- [In Defense of Classical Image Processing: Fast Depth Completion on the CPU](https://mlsanity.com/paper/1802.00036/) (2018)
- [NodeSLAM: Neural Object Descriptors for Multi-View Shape Reconstruction](https://mlsanity.com/paper/2004.04485/) (2020)
- [SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representations](https://mlsanity.com/paper/1903.06482/) (2019)
- [Meaningful Maps With Object-Oriented Semantic Mapping](https://mlsanity.com/paper/1609.07849/) (2016)
- [Incremental Class Discovery for Semantic Segmentation with RGBD Sensing](https://mlsanity.com/paper/1907.10008/) (2019)
- [SemanticPaint: A Framework for the Interactive Segmentation of 3D Scenes](https://mlsanity.com/paper/1510.03727/) (2015)
- [A Robust 3D-2D Interactive Tool for Scene Segmentation and Annotation](https://mlsanity.com/paper/1610.05883/) (2016)
- [Hierarchical Pyramid Representations for Semantic Segmentation](https://mlsanity.com/paper/2104.01792/) (2021)

## Built on by (most cited first, 28 of 28)

- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations](https://mlsanity.com/paper/2209.03494/) (2022)
- [CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory](https://mlsanity.com/paper/2210.05663/) (2022)
- [SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields](https://mlsanity.com/paper/2211.12254/) (2022)
- [DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images](https://mlsanity.com/paper/2208.07227/) (2022)
- [vMAP: Vectorised Object Mapping for Neural Field SLAM](https://mlsanity.com/paper/2302.01838/) (2023)
- [LaTeRF: Label and Text Driven Object Radiance Fields](https://mlsanity.com/paper/2207.01583/) (2022)
- [Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understanding](https://mlsanity.com/paper/2210.03043/) (2022)
- [Interactive Object Segmentation in 3D Point Clouds](https://mlsanity.com/paper/2204.07183/) (2022)
- [VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations](https://mlsanity.com/paper/2305.12427/) (2023)
- [Unsupervised Continual Semantic Adaptation through Neural Rendering](https://mlsanity.com/paper/2211.13969/) (2022)
- [Multi-Object Navigation with dynamically learned neural implicit representations](https://mlsanity.com/paper/2210.05129/) (2022)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectories](https://mlsanity.com/paper/2311.12174/) (2023)
- [AutoNeRF: Training Implicit Scene Representations with Autonomous Agents](https://mlsanity.com/paper/2304.11241/) (2023)
- [Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Review](https://mlsanity.com/paper/2402.11141/) (2024)
- [Neural Radiance Fields: Past, Present, and Future](https://mlsanity.com/paper/2304.10050/) (2023)
- [Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps](https://mlsanity.com/paper/2209.12744/) (2022)
- [WildSeg3D: Segment Any 3D Objects in the Wild from 2D Images](https://mlsanity.com/paper/2503.08407/) (2025)
- [Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations](https://mlsanity.com/paper/2310.17880/) (2023)
- [Semantic Is Enough: Only Semantic Information For NeRF Reconstruction](https://mlsanity.com/paper/2403.16043/) (2024)
- [Self-supervised Learning of Neural Implicit Feature Fields for Camera Pose Refinement](https://mlsanity.com/paper/2406.08463/) (2024)
- [Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter](https://mlsanity.com/paper/2210.17325/) (2022)
- [Factored Neural Representation for Scene Understanding](https://mlsanity.com/paper/2304.10950/) (2023)
- [You Only Label Once: 3D Box Adaptation from Point Cloud to Image via Semi-Supervised Learning](https://mlsanity.com/paper/2211.09302/) (2022)
- [LiLMaps: Learnable Implicit Language Maps](https://mlsanity.com/paper/2501.03304/) (2025)
- [SemanticFeels: Semantic Labeling during In-Hand Manipulation](https://mlsanity.com/paper/2602.14099/) (2026)

## Abstract (arXiv)

Joint representation of geometry, colour and semantics using a 3D neural field enables accurate dense labelling from ultra-sparse interactions as a user reconstructs a scene in real-time using a handheld RGB-D sensor. Our iLabel system requires no training data, yet can densely label scenes more accurately than standard methods trained on large, expensively labelled image datasets. Furthermore, it works in an 'open set' manner, with semantic classes defined on the fly by the user. ILabel's underlying model is a multilayer perceptron (MLP) trained from scratch in real-time to learn a joint neural scene representation. The scene model is updated and visualised in real-time, allowing the user to focus interactions to achieve efficient labelling. A room or similar scene can be accurately labelled into 10+ semantic categories with only a few tens of clicks. Quantitative labelling accuracy scales powerfully with the number of clicks, and rapidly surpasses standard pre-trained semantic segmentation methods. We also demonstrate a hierarchical labelling variant.