# Generating Long Videos of Dynamic Scenes

arXiv: https://arxiv.org/abs/2206.03429 · HTML: https://mlsanity.com/paper/2206.03429/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 70 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 30 of 55)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://mlsanity.com/paper/2204.03638/) (2022)
- [Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image](https://mlsanity.com/paper/2012.09855/) (2020)

## Built on by (most cited first, 30 of 113)

- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [MAGVIT: Masked Generative Video Transformer](https://mlsanity.com/paper/2212.05199/) (2022)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving](https://mlsanity.com/paper/2311.17918/) (2023)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling](https://mlsanity.com/paper/2310.15169/) (2023)
- [Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation](https://mlsanity.com/paper/2410.05363/) (2024)
- [GenAD: Generalized Predictive Model for Autonomous Driving](https://mlsanity.com/paper/2403.09630/) (2024)
- [ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation](https://mlsanity.com/paper/2402.04324/) (2024)
- [OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models](https://mlsanity.com/paper/2502.01061/) (2025)
- [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023)
- [Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis](https://mlsanity.com/paper/2402.14797/) (2024)
- [Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency](https://mlsanity.com/paper/2409.02634/) (2024)
- [SimDA: Simple Diffusion Adapter for Efficient Video Generation](https://mlsanity.com/paper/2308.09710/) (2023)
- [Loong: Generating Minute-level Long Videos with Autoregressive Language Models](https://mlsanity.com/paper/2410.02757/) (2024)
- [FreeInit: Bridging Initialization Gap in Video Diffusion Models](https://mlsanity.com/paper/2312.07537/) (2023)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation](https://mlsanity.com/paper/2506.09350/) (2025)
- [FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention](https://mlsanity.com/paper/2407.19918/) (2024)
- [LLM-grounded Video Diffusion Models](https://mlsanity.com/paper/2309.17444/) (2023)

## Abstract (arXiv)

We present a video generation model that accurately reproduces object motion, changes in camera viewpoint, and new content that arises over time. Existing video generation methods often fail to produce new content as a function of time while maintaining consistencies expected in real environments, such as plausible dynamics and object persistence. A common failure case is for content to never change due to over-reliance on inductive biases to provide temporal consistency, such as a single latent code that dictates content for the entire video. On the other extreme, without long-term consistency, generated videos may morph unrealistically between different scenes. To address these limitations, we prioritize the time axis by redesigning the temporal latent representation and learning long-term consistency from data by training on longer videos. To this end, we leverage a two-phase training strategy, where we separately train using longer videos at a low resolution and shorter videos at a high resolution. To evaluate the capabilities of our model, we introduce two new benchmark datasets with explicit focus on long-term temporal dynamics.