# Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis

arXiv: https://arxiv.org/abs/2102.13329 · HTML: https://mlsanity.com/paper/2102.13329/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 43 references that are in the map.
- First posted on arXiv 26 Feb 2021.

## Builds on (most cited first, 30 of 36)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1808.00948/) (2018)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [Temporal Generative Adversarial Nets with Singular Value Clipping](https://mlsanity.com/paper/1611.06624/) (2016)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017)
- [Self-Supervised Visual Planning with Temporal Skip Connections](https://mlsanity.com/paper/1710.05268/) (2017)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [Recycle-GAN: Unsupervised Video Retargeting](https://mlsanity.com/paper/1808.05174/) (2018)
- [Learning to Generate Long-term Future via Hierarchical Prediction](https://mlsanity.com/paper/1704.05831/) (2017)
- [Neural Kinematic Networks for Unsupervised Motion Retargetting](https://mlsanity.com/paper/1804.05653/) (2018)
- [Flow-Grounded Spatial-Temporal Video Prediction from Still Images](https://mlsanity.com/paper/1807.09755/) (2018)
- [Video Generation from Single Semantic Label Map](https://mlsanity.com/paper/1903.04480/) (2019)
- [FLNet: Landmark Driven Fetching and Learning Network for Faithful Talking Facial Animation Synthesis](https://mlsanity.com/paper/1911.09224/) (2019)

## Built on by (most cited first, 1 of 1)

- [FitVid: Overfitting in Pixel-Level Video Prediction](https://mlsanity.com/paper/2106.13195/) (2021)

## Abstract (arXiv)

Generating videos with content and motion variations is a challenging task in computer vision. While the recent development of GAN allows video generation from latent representations, it is not easy to produce videos with particular content of motion patterns of interest. In this paper, we propose Dual Motion Transfer GAN (Dual-MTGAN), which takes image and video data as inputs while learning disentangled content and motion representations. Our Dual-MTGAN is able to perform deterministic motion transfer and stochastic motion generation. Based on a given image, the former preserves the input content and transfers motion patterns observed from another video sequence, and the latter directly produces videos with plausible yet diverse motion patterns based on the input image. The proposed model is trained in an end-to-end manner, without the need to utilize pre-defined motion features like pose or facial landmarks. Our quantitative and qualitative results would confirm the effectiveness and robustness of our model in addressing such conditioned image-to-video tasks.