# Deep Video-Based Performance Cloning

arXiv: https://arxiv.org/abs/1808.06847 · HTML: https://mlsanity.com/paper/1808.06847/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2018.

## Built on by (most cited first, 30 of 32)

- [Editable Free-viewpoint Video Using a Layered Neural Representation](https://mlsanity.com/paper/2104.14786/) (2021)
- [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021)
- [Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control](https://mlsanity.com/paper/2106.02019/) (2021)
- [Driving-Signal Aware Full-Body Avatars](https://mlsanity.com/paper/2105.10441/) (2021)
- [Modeling Clothing as a Separate Layer for an Animatable Human Avatar](https://mlsanity.com/paper/2106.14879/) (2021)
- [Style and Pose Control for Image Synthesis of Humans from a Single Monocular View](https://mlsanity.com/paper/2102.11263/) (2021)
- [Putting People in Their Place: Affordance-Aware Human Insertion into Scenes](https://mlsanity.com/paper/2304.14406/) (2023)
- [HumanGAN: A Generative Model of Humans Images](https://mlsanity.com/paper/2103.06902/) (2021)
- [Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN](https://mlsanity.com/paper/2109.06166/) (2021)
- [Single-Image 3D Human Digitization with Shape-Guided Diffusion](https://mlsanity.com/paper/2311.09221/) (2023)
- [HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars](https://mlsanity.com/paper/2112.10203/) (2021)
- [Dynamic Neural Garments](https://mlsanity.com/paper/2102.11811/) (2021)
- [Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis](https://mlsanity.com/paper/2111.05916/) (2021)
- [Hallucinating Pose-Compatible Scenes](https://mlsanity.com/paper/2112.06909/) (2021)
- [EgoRenderer: Rendering Human Avatars from Egocentric Camera Images](https://mlsanity.com/paper/2111.12685/) (2021)
- [MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks](https://mlsanity.com/paper/2112.10082/) (2021)
- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [Flow Guided Transformable Bottleneck Networks for Motion Retargeting](https://mlsanity.com/paper/2106.07771/) (2021)
- [HDHumans: A Hybrid Approach for High-fidelity Digital Humans](https://mlsanity.com/paper/2210.12003/) (2022)
- [Real-Time Neural Character Rendering with Pose-Guided Multiplane Images](https://mlsanity.com/paper/2204.11820/) (2022)
- [SurMo: Surface-based 4D Motion Modeling for Dynamic Human Rendering](https://mlsanity.com/paper/2404.01225/) (2024)
- [A Shape-Aware Retargeting Approach to Transfer Human Motion and Appearance in Monocular Videos](https://mlsanity.com/paper/2103.15596/) (2021)
- [Pose-to-Motion: Cross-Domain Motion Retargeting with Pose Prior](https://mlsanity.com/paper/2310.20249/) (2023)
- [TemporalUV: Capturing Loose Clothing with Temporally Coherent UV Coordinates](https://mlsanity.com/paper/2204.03671/) (2022)
- [A Novel Application of Image-to-Image Translation: Chromosome Straightening Framework by Learning from a Single Image](https://mlsanity.com/paper/2103.02835/) (2021)
- [JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting](https://mlsanity.com/paper/2106.09679/) (2021)
- [Text-Conditional Contextualized Avatars For Zero-Shot Personalization](https://mlsanity.com/paper/2304.07410/) (2023)
- [Neural Sign Reenactor: Deep Photorealistic Sign Language Retargeting](https://mlsanity.com/paper/2209.01470/) (2022)
- [Audio Input Generates Continuous Frames to Synthesize Facial Video Using Generative Adiversarial Networks](https://mlsanity.com/paper/2207.08813/) (2022)
- [Muscles in Action](https://mlsanity.com/paper/2212.02978/) (2022)

## Abstract (arXiv)

We present a new video-based performance cloning technique. After training a deep generative network using a reference video capturing the appearance and dynamics of a target actor, we are able to generate videos where this actor reenacts other performances. All of the training data and the driving performances are provided as ordinary video segments, without motion capture or depth information. Our generative model is realized as a deep neural network with two branches, both of which train the same space-time conditional generator, using shared weights. One branch, responsible for learning to generate the appearance of the target actor in various poses, uses paired training data, self-generated from the reference video. The second branch uses unpaired data to improve generation of temporally coherent video renditions of unseen pose sequences. We demonstrate a variety of promising results, where our method is able to generate temporally coherent videos, for challenging scenarios where the reference and driving videos consist of very different dance performances. Supplementary video: https://youtu.be/JpwsEeqNhhA.