# Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation

arXiv: https://arxiv.org/abs/2001.04947 · HTML: https://mlsanity.com/paper/2001.04947/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2020.

## Built on by (most cited first, 13 of 13)

- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Neural Rays for Occlusion-aware Image-based Rendering](https://mlsanity.com/paper/2107.13421/) (2021)
- [AvatarReX: Real-time Expressive Full-body Avatars](https://mlsanity.com/paper/2305.04789/) (2023)
- [Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos](https://mlsanity.com/paper/2203.08133/) (2022)
- [SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image](https://mlsanity.com/paper/2309.01855/) (2023)
- [Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis](https://mlsanity.com/paper/2111.05916/) (2021)
- [NDF: Neural Deformable Fields for Dynamic Human Modelling](https://mlsanity.com/paper/2207.09193/) (2022)
- [EgoRenderer: Rendering Human Avatars from Egocentric Camera Images](https://mlsanity.com/paper/2111.12685/) (2021)
- [HDhuman: High-quality Human Novel-view Rendering from Sparse Views](https://mlsanity.com/paper/2201.08158/) (2022)
- [Text-Conditional Contextualized Avatars For Zero-Shot Personalization](https://mlsanity.com/paper/2304.07410/) (2023)
- [Animatable Virtual Humans: Learning pose-dependent human representations in UV space for interactive performance synthesis](https://mlsanity.com/paper/2310.03615/) (2023)
- [Robust Pose Transfer with Dynamic Details using Neural Video Rendering](https://mlsanity.com/paper/2106.14132/) (2021)
- [Shape and Reflectance Reconstruction in Uncontrolled Environments by Differentiable Rendering](https://mlsanity.com/paper/2110.12975/) (2021)

## Abstract (arXiv)

Synthesizing realistic videos of humans using neural networks has been a popular alternative to the conventional graphics-based rendering pipeline due to its high efficiency. Existing works typically formulate this as an image-to-image translation problem in 2D screen space, which leads to artifacts such as over-smoothing, missing body parts, and temporal instability of fine-scale detail, such as pose-dependent wrinkles in the clothing. In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. More specifically, our method relies on the combination of two convolutional neural networks (CNNs). Given the pose information, the first CNN predicts a dynamic texture map that contains time-coherent high-frequency details, and the second CNN conditions the generation of the final video on the temporally coherent output of the first CNN. We demonstrate several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.