# Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample

arXiv: https://arxiv.org/abs/2006.12226 · HTML: https://mlsanity.com/paper/2006.12226/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2020.

## Built on by (most cited first, 30 of 43)

- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising](https://mlsanity.com/paper/2305.18264/) (2023)
- [A Hierarchical Transformation-Discriminating Generative Model for Few Shot Anomaly Detection](https://mlsanity.com/paper/2104.14535/) (2021)
- [SinDDM: A Single Image Denoising Diffusion Model](https://mlsanity.com/paper/2211.16582/) (2022)
- [SinFusion: Training Diffusion Models on a Single Image or Video](https://mlsanity.com/paper/2211.11743/) (2022)
- [Drop the GAN: In Defense of Patches Nearest Neighbors as Single Image Generative Models](https://mlsanity.com/paper/2103.15545/) (2021)
- [VMBench: A Benchmark for Perception-Aligned Video Motion Generation](https://mlsanity.com/paper/2503.10076/) (2025)
- [Towards Consistent Video Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.17431/) (2023)
- [Text2Light: Zero-Shot Text-Driven HDR Panorama Generation](https://mlsanity.com/paper/2209.09898/) (2022)
- [Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive](https://mlsanity.com/paper/2401.08815/) (2024)
- [Catch-A-Waveform: Learning to Generate Audio from a Single Short Example](https://mlsanity.com/paper/2106.06426/) (2021)
- [FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance](https://mlsanity.com/paper/2408.08189/) (2024)
- [Strumming to the Beat: Audio-Conditioned Contrastive Video Textures](https://mlsanity.com/paper/2104.02687/) (2021)
- [PetsGAN: Rethinking Priors for Single Image Generation](https://mlsanity.com/paper/2203.01488/) (2022)
- [Diverse Generation from a Single Video Made Possible](https://mlsanity.com/paper/2109.08591/) (2021)
- [ExSinGAN: Learning an Explainable Generative Model from a Single Image](https://mlsanity.com/paper/2105.07350/) (2021)
- [A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction](https://mlsanity.com/paper/2110.03446/) (2021)
- [Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generation](https://mlsanity.com/paper/2503.24379/) (2025)
- [MOGAN: Morphologic-structure-aware Generative Learning from a Single Image](https://mlsanity.com/paper/2103.02997/) (2021)
- [Meta Internal Learning](https://mlsanity.com/paper/2110.02900/) (2021)
- [Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generation](https://mlsanity.com/paper/2412.18688/) (2024)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [Segment Anything for Videos: A Systematic Survey](https://mlsanity.com/paper/2408.08315/) (2024)
- [Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applications](https://mlsanity.com/paper/2510.21887/) (2025)
- [One-Shot Synthesis of Images and Segmentation Masks](https://mlsanity.com/paper/2209.07547/) (2022)
- [JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting](https://mlsanity.com/paper/2106.09679/) (2021)
- [Diverse Single Image Generation with Controllable Global Structure](https://mlsanity.com/paper/2102.04780/) (2021)

## Abstract (arXiv)

We consider the task of generating diverse and novel videos from a single video sample. Recently, new hierarchical patch-GAN based approaches were proposed for generating diverse images, given only a single sample at training time. Moving to videos, these approaches fail to generate diverse samples, and often collapse into generating samples similar to the training video. We introduce a novel patch-based variational autoencoder (VAE) which allows for a much greater diversity in generation. Using this tool, a new hierarchical video generation scheme is constructed: at coarse scales, our patch-VAE is employed, ensuring samples are of high diversity. Subsequently, at finer scales, a patch-GAN renders the fine details, resulting in high quality videos. Our experiments show that the proposed method produces diverse samples in both the image domain, and the more challenging video domain.