# FutureGAN: Anticipating the Future Frames of Video Sequences using Spatio-Temporal 3d Convolutions in Progressively Growing GANs

arXiv: https://arxiv.org/abs/1810.01325 · HTML: https://mlsanity.com/paper/1810.01325/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2018.

## Built on by (most cited first, 24 of 24)

- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [Reinforcement Learning with Action-Free Pre-Training from Videos](https://mlsanity.com/paper/2203.13880/) (2022)
- [OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning](https://mlsanity.com/paper/2306.11249/) (2023)
- [Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks](https://mlsanity.com/paper/2110.04076/) (2021)
- [HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator](https://mlsanity.com/paper/2209.07143/) (2022)
- [MIMO Is All You Need : A Strong Multi-In-Multi-Out Baseline for Video Prediction](https://mlsanity.com/paper/2212.04655/) (2022)
- [Implicit Stacked Autoregressive Model for Video Prediction](https://mlsanity.com/paper/2303.07849/) (2023)
- [Diverse Generation from a Single Video Made Possible](https://mlsanity.com/paper/2109.08591/) (2021)
- [A review of Generative Adversarial Networks (GANs) and its applications in a wide variety of disciplines -- From Medical to Remote Sensing](https://mlsanity.com/paper/2110.01442/) (2021)
- [Pix2Gif: Motion-Guided Diffusion for GIF Generation](https://mlsanity.com/paper/2403.04634/) (2024)
- [Conditional COT-GAN for Video Prediction with Kernel Smoothing](https://mlsanity.com/paper/2106.05658/) (2021)
- [Belief Propagation as Diffusion](https://mlsanity.com/paper/2107.12230/) (2021)
- [Pyramidal Predictive Network: A Model for Visual-frame Prediction Based on Predictive Coding Theory](https://mlsanity.com/paper/2208.07021/) (2022)
- [Predictive Coding Based Multiscale Network with Encoder-Decoder LSTM for Video Prediction](https://mlsanity.com/paper/2212.11642/) (2022)
- [TAFormer: A Unified Target-Aware Transformer for Video and Motion Joint Prediction in Aerial Scenes](https://mlsanity.com/paper/2403.18238/) (2024)
- [Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling](https://mlsanity.com/paper/2504.05537/) (2025)
- [Masked Autoregressive Model for Weather Forecasting](https://mlsanity.com/paper/2409.20117/) (2024)
- [FREGAN : an application of generative adversarial networks in enhancing the frame rate of videos](https://mlsanity.com/paper/2111.01105/) (2021)
- [Diverse Video Generation from a Single Video](https://mlsanity.com/paper/2205.05725/) (2022)
- [Investigating Low Data, Confidence Aware Image Prediction on Smooth Repetitive Videos using Gaussian Processes](https://mlsanity.com/paper/2307.11259/) (2023)
- [SIAM: A Simple Alternating Mixer for Video Prediction](https://mlsanity.com/paper/2311.11683/) (2023)
- [CausalVE: Face Video Privacy Encryption via Causal Video Prediction](https://mlsanity.com/paper/2409.19306/) (2024)

## Abstract (arXiv)

We introduce a new encoder-decoder GAN model, FutureGAN, that predicts future frames of a video sequence conditioned on a sequence of past frames. During training, the networks solely receive the raw pixel values as an input, without relying on additional constraints or dataset specific conditions. To capture both the spatial and temporal components of a video sequence, spatio-temporal 3d convolutions are used in all encoder and decoder modules. Further, we utilize concepts of the existing progressively growing GAN (PGGAN) that achieves high-quality results on generating high-resolution single images. The FutureGAN model extends this concept to the complex task of video prediction. We conducted experiments on three different datasets, MovingMNIST, KTH Action, and Cityscapes. Our results show that the model learned representations to transform the information of an input sequence into a plausible future sequence effectively for all three datasets. The main advantage of the FutureGAN framework is that it is applicable to various different datasets without additional changes, whilst achieving stable results that are competitive to the state-of-the-art in video prediction. Our code is available at https://github.com/TUM-LMF/FutureGAN.