# Semantic Implicit Neural Scene Representations With Semi-Supervised Training

arXiv: https://arxiv.org/abs/2003.12673 · HTML: https://mlsanity.com/paper/2003.12673/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2020.

## Built on by (most cited first, 9 of 9)

- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation](https://mlsanity.com/paper/2201.07786/) (2022)
- [A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation](https://mlsanity.com/paper/2104.07645/) (2021)
- [Fast Training of Neural Lumigraph Representations using Meta Learning](https://mlsanity.com/paper/2106.14942/) (2021)
- [Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild](https://mlsanity.com/paper/2111.12728/) (2021)
- [Label Distribution Learning via Implicit Distribution Representation](https://mlsanity.com/paper/2209.13824/) (2022)
- [MADS: Modulated Auto-Decoding SIREN for time series imputation](https://mlsanity.com/paper/2307.00868/) (2023)
- [Towards Generalising Neural Implicit Representations](https://mlsanity.com/paper/2101.12690/) (2021)
- [Focused Specific Objects NeRF](https://mlsanity.com/paper/2308.05970/) (2023)

## Abstract (arXiv)

The recent success of implicit neural scene representations has presented a viable new method for how we capture and store 3D scenes. Unlike conventional 3D representations, such as point clouds, which explicitly store scene properties in discrete, localized units, these implicit representations encode a scene in the weights of a neural network which can be queried at any coordinate to produce these same scene properties. Thus far, implicit representations have primarily been optimized to estimate only the appearance and/or 3D geometry information in a scene. We take the next step and demonstrate that an existing implicit representation (SRNs) is actually multi-modal; it can be further leveraged to perform per-point semantic segmentation while retaining its ability to represent appearance and geometry. To achieve this multi-modal behavior, we utilize a semi-supervised learning strategy atop the existing pre-trained scene representation. Our method is simple, general, and only requires a few tens of labeled 2D segmentation masks in order to achieve dense 3D semantic segmentation. We explore two novel applications for this semantically aware implicit neural scene representation: 3D novel view and semantic label synthesis given only a single input RGB image or 2D label mask, as well as 3D interpolation of appearance and semantics.