# Diverse Video Generation from a Single Video

arXiv: https://arxiv.org/abs/2205.05725 · HTML: https://mlsanity.com/paper/2205.05725/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 30 of 32)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [Temporal Generative Adversarial Nets with Singular Value Clipping](https://mlsanity.com/paper/1611.06624/) (2016)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [Delving Deeper into Convolutional Networks for Learning Video Representations](https://mlsanity.com/paper/1511.06432/) (2015)
- [Visual Attribute Transfer through Deep Image Analogy](https://mlsanity.com/paper/1705.01088/) (2017)
- [Stochastic Latent Residual Video Prediction](https://mlsanity.com/paper/2002.09219/) (2020)
- [Unpaired Motion Style Transfer from Video to Animation](https://mlsanity.com/paper/2005.05751/) (2020)
- [Recycle-GAN: Unsupervised Video Retargeting](https://mlsanity.com/paper/1808.05174/) (2018)
- [High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks](https://mlsanity.com/paper/1911.01655/) (2019)
- [World-Consistent Video-to-Video Synthesis](https://mlsanity.com/paper/2007.08509/) (2020)
- [Drop the GAN: In Defense of Patches Nearest Neighbors as Single Image Generative Models](https://mlsanity.com/paper/2103.15545/) (2021)
- [Hierarchical Long-term Video Prediction without Supervision](https://mlsanity.com/paper/1806.04768/) (2018)
- [Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample](https://mlsanity.com/paper/2006.12226/) (2020)
- [TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting](https://mlsanity.com/paper/2003.14401/) (2020)
- [STCN: Stochastic Temporal Convolutional Networks](https://mlsanity.com/paper/1902.06568/) (2019)
- [FutureGAN: Anticipating the Future Frames of Video Sequences using Spatio-Temporal 3d Convolutions in Progressively Growing GANs](https://mlsanity.com/paper/1810.01325/) (2018)
- [InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation](https://mlsanity.com/paper/2101.03049/) (2021)
- [Human Motion Transfer from Poses in the Wild](https://mlsanity.com/paper/2004.03142/) (2020)

## Abstract (arXiv)

GANs are able to perform generation and manipulation tasks, trained on a single video. However, these single video GANs require unreasonable amount of time to train on a single video, rendering them almost impractical. In this paper we question the necessity of a GAN for generation from a single video, and introduce a non-parametric baseline for a variety of generation and manipulation tasks. We revive classical space-time patches-nearest-neighbors approaches and adapt them to a scalable unconditional generative model, without any learning. This simple baseline surprisingly outperforms single-video GANs in visual quality and realism (confirmed by quantitative and qualitative evaluations), and is disproportionately faster (runtime reduced from several days to seconds). Our approach is easily scaled to Full-HD videos. We also use the same framework to demonstrate video analogies and spatio-temporal retargeting. These observations show that classical approaches significantly outperform heavy deep learning machinery for these tasks. This sets a new baseline for single-video generation and manipulation tasks, and no less important -- makes diverse generation from a single video practically possible for the first time.