# SSDNeRF: Semantic Soft Decomposition of Neural Radiance Fields

arXiv: https://arxiv.org/abs/2212.03406 · HTML: https://mlsanity.com/paper/2212.03406/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 7 Dec 2022.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [DeRF: Decomposed Radiance Fields](https://mlsanity.com/paper/2011.12490/) (2020)
- [Unsupervised Discovery of Object Radiance Fields](https://mlsanity.com/paper/2107.07905/) (2021)
- [Object-Centric Neural Scene Rendering](https://mlsanity.com/paper/2012.08503/) (2020)
- [Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation](https://mlsanity.com/paper/2104.01148/) (2021)
- [T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis](https://mlsanity.com/paper/2109.15271/) (2021)
- [Learning Compositional Radiance Fields of Dynamic Human Heads](https://mlsanity.com/paper/2012.09955/) (2020)
- [Neural Volume Rendering: NeRF And Beyond](https://mlsanity.com/paper/2101.05204/) (2020)

## Built on by (most cited first, 9 of 9)

- [Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models](https://mlsanity.com/paper/2310.03020/) (2023)
- [Text-Guided Generation and Editing of Compositional 3D Avatars](https://mlsanity.com/paper/2309.07125/) (2023)
- [TELA: Text to Layer-wise 3D Clothed Human Generation](https://mlsanity.com/paper/2404.16748/) (2024)
- [PEGASUS: Personalized Generative 3D Avatars with Composable Attributes](https://mlsanity.com/paper/2402.10636/) (2024)
- [Neural Assets: Volumetric Object Capture and Rendering for Interactive Environments](https://mlsanity.com/paper/2212.06125/) (2022)
- [LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset](https://mlsanity.com/paper/2312.12418/) (2023)
- [Part-aware Shape Generation with Latent 3D Diffusion of Neural Voxel Fields](https://mlsanity.com/paper/2405.00998/) (2024)
- [DORec: Decomposed Object Reconstruction and Segmentation Utilizing 2D Self-Supervised Features](https://mlsanity.com/paper/2310.11092/) (2023)
- [How Far Can INRs Go? Cross-Domain Parameter-efficient INR-Based Semantic Segmentation for Brain MRI](https://mlsanity.com/paper/2609.31573/) (2026)

## Abstract (arXiv)

Neural Radiance Fields (NeRFs) encode the radiance in a scene parameterized by the scene's plenoptic function. This is achieved by using an MLP together with a mapping to a higher-dimensional space, and has been proven to capture scenes with a great level of detail. Naturally, the same parameterization can be used to encode additional properties of the scene, beyond just its radiance. A particularly interesting property in this regard is the semantic decomposition of the scene. We introduce a novel technique for semantic soft decomposition of neural radiance fields (named SSDNeRF) which jointly encodes semantic signals in combination with radiance signals of a scene. Our approach provides a soft decomposition of the scene into semantic parts, enabling us to correctly encode multiple semantic classes blending along the same direction -- an impossible feat for existing methods. Not only does this lead to a detailed, 3D semantic representation of the scene, but we also show that the regularizing effects of the MLP used for encoding help to improve the semantic representation. We show state-of-the-art segmentation and reconstruction results on a dataset of common objects and demonstrate how the proposed approach can be applied for high quality temporally consistent video editing and re-compositing on a dataset of casually captured selfie videos.