# NormalGAN: Learning Detailed 3D Human from a Single RGB-D Image

arXiv: https://arxiv.org/abs/2007.15340 · HTML: https://mlsanity.com/paper/2007.15340/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2020.

## Built on by (most cited first, 24 of 24)

- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors](https://mlsanity.com/paper/2105.01859/) (2021)
- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)
- [NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Views](https://mlsanity.com/paper/2211.16431/) (2022)
- [Structured Local Radiance Fields for Human Avatar Modeling](https://mlsanity.com/paper/2203.14478/) (2022)
- [gDNA: Towards Generative Detailed Neural Avatars](https://mlsanity.com/paper/2201.04123/) (2022)
- [DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras](https://mlsanity.com/paper/2207.08000/) (2022)
- [Single-Image 3D Human Digitization with Shape-Guided Diffusion](https://mlsanity.com/paper/2311.09221/) (2023)
- [M3D-VTON: A Monocular-to-3D Virtual Try-On Network](https://mlsanity.com/paper/2108.05126/) (2021)
- [Human Performance Capture from Monocular Video in the Wild](https://mlsanity.com/paper/2111.14672/) (2021)
- [AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture](https://mlsanity.com/paper/2207.02031/) (2022)
- [Self-supervised 3D Representation Learning of Dressed Humans from Social Media Videos](https://mlsanity.com/paper/2103.03319/) (2021)
- [Layered-Garment Net: Generating Multiple Implicit Garment Layers from a Single Image](https://mlsanity.com/paper/2211.11931/) (2022)
- [ANIM: Accurate Neural Implicit Model for Human Reconstruction from a single RGB-D image](https://mlsanity.com/paper/2403.10357/) (2024)
- [Human as Points: Explicit Point-based 3D Human Reconstruction from Single-view RGB Images](https://mlsanity.com/paper/2311.02892/) (2023)
- [Advances in Radiance Field for Dynamic Scene: From Neural Field to Gaussian Field](https://mlsanity.com/paper/2505.10049/) (2025)
- [Geometry-aware Two-scale PIFu Representation for Human Reconstruction](https://mlsanity.com/paper/2112.02082/) (2021)
- [Occupancy Planes for Single-view RGB-D Human Reconstruction](https://mlsanity.com/paper/2208.02817/) (2022)
- [PIFu for the Real World: A Self-supervised Framework to Reconstruct Dressed Human from Single-view Images](https://mlsanity.com/paper/2208.10769/) (2022)
- [Monocular Human Digitization via Implicit Re-projection Networks](https://mlsanity.com/paper/2205.06468/) (2022)
- [DI-Net : Decomposed Implicit Garment Transfer Network for Digital Clothed 3D Human](https://mlsanity.com/paper/2311.16818/) (2023)
- [Pixel2ISDF: Implicit Signed Distance Fields based Human Body Model from Multi-view and Multi-pose Images](https://mlsanity.com/paper/2212.02765/) (2022)
- [InstantGeoAvatar: Effective Geometry and Appearance Modeling of Animatable Avatars from Monocular Video](https://mlsanity.com/paper/2411.01512/) (2024)

## Abstract (arXiv)

We propose NormalGAN, a fast adversarial learning-based method to reconstruct the complete and detailed 3D human from a single RGB-D image. Given a single front-view RGB-D image, NormalGAN performs two steps: front-view RGB-D rectification and back-view RGBD inference. The final model was then generated by simply combining the front-view and back-view RGB-D information. However, inferring backview RGB-D image with high-quality geometric details and plausible texture is not trivial. Our key observation is: Normal maps generally encode much more information of 3D surface details than RGB and depth images. Therefore, learning geometric details from normal maps is superior than other representations. In NormalGAN, an adversarial learning framework conditioned by normal maps is introduced, which is used to not only improve the front-view depth denoising performance, but also infer the back-view depth image with surprisingly geometric details. Moreover, for texture recovery, we remove shading information from the front-view RGB image based on the refined normal map, which further improves the quality of the back-view color inference. Results and experiments on both testing data set and real captured data demonstrate the superior performance of our approach. Given a consumer RGB-D sensor, NormalGAN can generate the complete and detailed 3D human reconstruction results in 20 fps, which further enables convenient interactive experiences in telepresence, AR/VR and gaming scenarios.