# Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis

arXiv: https://arxiv.org/abs/2111.05916 · HTML: https://mlsanity.com/paper/2111.05916/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 10 Nov 2021.

## Builds on (most cited first, 30 of 35)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [StyleRig: Rigging StyleGAN for 3D Control over Portrait Images](https://mlsanity.com/paper/2004.00121/) (2020)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [Deformable GANs for Pose-based Human Image Generation](https://mlsanity.com/paper/1801.00055/) (2017)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)
- [Progressive Pose Attention Transfer for Person Image Generation](https://mlsanity.com/paper/1904.03349/) (2019)
- [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021)
- [Disentangled Person Image Generation](https://mlsanity.com/paper/1712.02621/) (2017)
- [Recycle-GAN: Unsupervised Video Retargeting](https://mlsanity.com/paper/1808.05174/) (2018)
- [Dense Pose Transfer](https://mlsanity.com/paper/1809.01995/) (2018)
- [SMPLpix: Neural Avatars from 3D Human Models](https://mlsanity.com/paper/2008.06872/) (2020)
- [Textured Neural Avatars](https://mlsanity.com/paper/1905.08776/) (2019)
- [StylePeople: A Generative Model of Fullbody Human Avatars](https://mlsanity.com/paper/2104.08363/) (2021)
- [Neural Re-Rendering of Humans from a Single Image](https://mlsanity.com/paper/2101.04104/) (2021)
- [Unsupervised Person Image Synthesis in Arbitrary Poses](https://mlsanity.com/paper/1809.10280/) (2018)
- [ANR: Articulated Neural Rendering for Virtual Avatars](https://mlsanity.com/paper/2012.12890/) (2020)
- [Style and Pose Control for Image Synthesis of Humans from a Single Monocular View](https://mlsanity.com/paper/2102.11263/) (2021)
- [Deep Video-Based Performance Cloning](https://mlsanity.com/paper/1808.06847/) (2018)

## Built on by (most cited first, 14 of 14)

- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [LEO: Generative Latent Image Animator for Human Video Synthesis](https://mlsanity.com/paper/2305.03989/) (2023)
- [LAC: Latent Action Composition for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2308.14500/) (2023)
- [Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans from a Single Camera](https://mlsanity.com/paper/2203.12780/) (2022)
- [RANA: Relightable Articulated Neural Avatars](https://mlsanity.com/paper/2212.03237/) (2022)
- [4DHumanOutfit: a multi-subject 4D dataset of human motion sequences in varying outfits exhibiting large displacements](https://mlsanity.com/paper/2306.07399/) (2023)
- [UniHuman: A Unified Model for Editing Human Images in the Wild](https://mlsanity.com/paper/2312.14985/) (2023)
- [Generalizable Neural Voxels for Fast Human Radiance Fields](https://mlsanity.com/paper/2303.15387/) (2023)
- [MoRF: Mobile Realistic Fullbody Avatars from a Monocular Video](https://mlsanity.com/paper/2303.10275/) (2023)
- [Normal-guided Garment UV Prediction for Human Re-texturing](https://mlsanity.com/paper/2303.06504/) (2023)
- [REMOT: A Region-to-Whole Framework for Realistic Human Motion Transfer](https://mlsanity.com/paper/2209.00475/) (2022)
- [Can We Find Neurons that Cause Unrealistic Images in Deep Generative Networks?](https://mlsanity.com/paper/2201.06346/) (2022)
- [Generalizable Neural Human Renderer](https://mlsanity.com/paper/2404.14199/) (2024)
- [Bidirectional Temporal Diffusion Model for Temporally Consistent Human Animation](https://mlsanity.com/paper/2307.00574/) (2023)

## Abstract (arXiv)

Synthesizing dynamic appearances of humans in motion plays a central role in applications such as AR/VR and video editing. While many recent methods have been proposed to tackle this problem, handling loose garments with complex textures and high dynamic motion still remains challenging. In this paper, we propose a video based appearance synthesis method that tackles such challenges and demonstrates high quality results for in-the-wild videos that have not been shown before. Specifically, we adopt a StyleGAN based architecture to the task of person specific video based motion retargeting. We introduce a novel motion signature that is used to modulate the generator weights to capture dynamic appearance changes as well as regularizing the single frame based pose estimates to improve temporal coherency. We evaluate our method on a set of challenging videos and show that our approach achieves state-of-the art performance both qualitatively and quantitatively.