# Do as we do: Multiple Person Video-To-Video Transfer

arXiv: https://arxiv.org/abs/2104.04721 · HTML: https://mlsanity.com/paper/2104.04721/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 10 Apr 2021.

## Builds on (most cited first, 27 of 27)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [Controllable Person Image Synthesis with Attribute-Decomposed GAN](https://mlsanity.com/paper/2003.12267/) (2020)
- [Attribute2Image: Conditional Image Generation from Visual Attributes](https://mlsanity.com/paper/1512.00570/) (2015)
- [Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World](https://mlsanity.com/paper/1803.08319/) (2018)
- [Be Your Own Prada: Fashion Synthesis with Structural Coherence](https://mlsanity.com/paper/1710.07346/) (2017)
- [ARID: A New Dataset for Recognizing Action in the Dark](https://mlsanity.com/paper/2006.03876/) (2020)
- [A Generative Model of People in Clothing](https://mlsanity.com/paper/1705.04098/) (2017)
- [Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds](https://mlsanity.com/paper/1802.01500/) (2018)
- [Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio](https://mlsanity.com/paper/2011.12999/) (2020)
- [Dance Dance Generation: Motion Transfer for Internet Videos](https://mlsanity.com/paper/1904.00129/) (2019)
- [Self-supervised Keypoint Correspondences for Multi-Person Pose Estimation and Tracking in Videos](https://mlsanity.com/paper/2004.12652/) (2020)
- [Human Motion Transfer from Poses in the Wild](https://mlsanity.com/paper/2004.03142/) (2020)
- [Human Pose Estimation for Real-World Crowded Scenarios](https://mlsanity.com/paper/1907.06922/) (2019)
- [Video synthesis of human upper body with realistic face](https://mlsanity.com/paper/1908.06607/) (2019)
- [Pose-Guided High-Resolution Appearance Transfer via Progressive Training](https://mlsanity.com/paper/2008.11898/) (2020)
- [Do As I Do: Transferring Human Motion and Appearance between Monocular Videos with Spatial and Temporal Constraints](https://mlsanity.com/paper/2001.02606/) (2020)
- [From Real to Synthetic and Back: Synthesizing Training Data for Multi-Person Scene Understanding](https://mlsanity.com/paper/2006.02110/) (2020)

## Built on by (most cited first, 1 of 1)

- [CTrGAN: Cycle Transformers GAN for Gait Transfer](https://mlsanity.com/paper/2206.15248/) (2022)

## Abstract (arXiv)

Our goal is to transfer the motion of real people from a source video to a target video with realistic results. While recent advances significantly improved image-to-image translations, only few works account for body motions and temporal consistency. However, those focus only on video re-targeting for a single actor/ for single actors. In this work, we propose a marker-less approach for multiple-person video-to-video transfer using pose as an intermediate representation. Given a source video with multiple persons dancing or working out, our method transfers the body motion of all actors to a new set of actors in a different video. Differently from recent "do as I do" methods, we focus specifically on transferring multiple person at the same time and tackle the related identity switch problem. Our method is able to convincingly transfer body motion to the target video, while preserving specific features of the target video, such as feet touching the floor and relative position of the actors. The evaluation is performed with visual quality and appearance metrics using publicly available videos with the permission of their owners.