# Semantic-Human: Neural Rendering of Humans from Monocular Video with Human Parsing

arXiv: https://arxiv.org/abs/2308.09894 · HTML: https://mlsanity.com/paper/2308.09894/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 19 Aug 2023.

## Builds on (most cited first, 30 of 32)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [NeuMan: Neural Human Radiance Field from a Single Video](https://mlsanity.com/paper/2203.12575/) (2022)
- [Robust Dynamic Radiance Fields](https://mlsanity.com/paper/2301.02239/) (2023)
- [Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering](https://mlsanity.com/paper/2109.07448/) (2021)
- [Instance-level Human Parsing via Part Grouping Network](https://mlsanity.com/paper/1808.00157/) (2018)
- [NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes](https://mlsanity.com/paper/2111.13260/) (2021)
- [Structured Local Radiance Fields for Human Avatar Modeling](https://mlsanity.com/paper/2203.14478/) (2022)
- [HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs](https://mlsanity.com/paper/2112.02789/) (2021)
- [SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field](https://mlsanity.com/paper/2303.13277/) (2023)
- [MonoHuman: Animatable Human Neural Field from Monocular Video](https://mlsanity.com/paper/2304.02001/) (2023)
- [CLA-NeRF: Category-Level Articulated Neural Radiance Field](https://mlsanity.com/paper/2202.00181/) (2022)
- [DoubleField: Bridging the Neural Surface and Radiance Fields for High-fidelity Human Reconstruction and Rendering](https://mlsanity.com/paper/2106.03798/) (2021)
- [Compressing Explicit Voxel Grid Representations: fast NeRFs become also small](https://mlsanity.com/paper/2210.12782/) (2022)
- [UV Volumes for Real-time Rendering of Editable Free-view Human Performance](https://mlsanity.com/paper/2203.14402/) (2022)
- [Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis](https://mlsanity.com/paper/2204.11798/) (2022)
- [Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild](https://mlsanity.com/paper/2012.12884/) (2020)
- [Semantic Ray: Learning a Generalizable Semantic Field with Cross-Reprojection Attention](https://mlsanity.com/paper/2303.13014/) (2023)
- [TriVol: Point Cloud Rendering via Triple Volumes](https://mlsanity.com/paper/2303.16485/) (2023)

## Built on by (most cited first, 1 of 1)

- [PoseGaussian: Pose-Driven Novel View Synthesis for Robust 3D Human Reconstruction](https://mlsanity.com/paper/2602.05190/) (2026)

## Abstract (arXiv)

The neural rendering of humans is a topic of great research significance. However, previous works mostly focus on achieving photorealistic details, neglecting the exploration of human parsing. Additionally, classical semantic work are all limited in their ability to efficiently represent fine results in complex motions. Human parsing is inherently related to radiance reconstruction, as similar appearance and geometry often correspond to similar semantic part. Furthermore, previous works often design a motion field that maps from the observation space to the canonical space, while it tends to exhibit either underfitting or overfitting, resulting in limited generalization. In this paper, we present Semantic-Human, a novel method that achieves both photorealistic details and viewpoint-consistent human parsing for the neural rendering of humans. Specifically, we extend neural radiance fields (NeRF) to jointly encode semantics, appearance and geometry to achieve accurate 2D semantic labels using noisy pseudo-label supervision. Leveraging the inherent consistency and smoothness properties of NeRF, Semantic-Human achieves consistent human parsing in both continuous and novel views. We also introduce constraints derived from the SMPL surface for the motion field and regularization for the recovered volumetric geometry. We have evaluated the model using the ZJU-MoCap dataset, and the obtained highly competitive results demonstrate the effectiveness of our proposed Semantic-Human. We also showcase various compelling applications, including label denoising, label synthesis and image editing, and empirically validate its advantageous properties.