# Stochastic Adversarial Video Prediction

arXiv: https://arxiv.org/abs/1804.01523 · HTML: https://mlsanity.com/paper/1804.01523/

## Key facts

- Cited by 173 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2018.

## Built on by (most cited first, 30 of 173)

- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion](https://mlsanity.com/paper/2111.12417/) (2021)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks](https://mlsanity.com/paper/2202.10571/) (2022)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [Structured World Models from Human Videos](https://mlsanity.com/paper/2308.10901/) (2023)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Video World Models with Long-term Spatial Memory](https://mlsanity.com/paper/2506.05284/) (2025)
- [Diffusion Models for Video Prediction and Infilling](https://mlsanity.com/paper/2206.07696/) (2022)
- [DriveGAN: Towards a Controllable High-Quality Neural Simulation](https://mlsanity.com/paper/2104.15060/) (2021)
- [MaskViT: Masked Visual Pre-Training for Video Prediction](https://mlsanity.com/paper/2206.11894/) (2022)
- [Generating Long Videos of Dynamic Scenes](https://mlsanity.com/paper/2206.03429/) (2022)
- [Reinforcement Learning with Action-Free Pre-Training from Videos](https://mlsanity.com/paper/2203.13880/) (2022)
- [Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis](https://mlsanity.com/paper/2402.14797/) (2024)
- [SimDA: Simple Diffusion Adapter for Efficient Video Generation](https://mlsanity.com/paper/2308.09710/) (2023)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [FitVid: Overfitting in Pixel-Level Video Prediction](https://mlsanity.com/paper/2106.13195/) (2021)
- [Dreamitate: Real-World Visuomotor Policy Learning via Video Generation](https://mlsanity.com/paper/2406.16862/) (2024)
- [Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.12126/) (2022)
- [PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation](https://mlsanity.com/paper/2601.03782/) (2026)

## Abstract (arXiv)

Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging -- the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior and concurrent work in these aspects.