mlsanityData to 7 Oct 2026

Diffusion Probabilistic Modeling for Video Generation

arXiv 2203.09481 ↗First posted 16 Mar 2022cs.CV cs.LG stat.MLcited by 237

On the map: Diffusion models → Video generation

Abstract (arXiv)

Denoising diffusion probabilistic models are a promising new class of generative models that mark a milestone in high-quality image generation. This paper showcases their ability to sequentially generate video, surpassing prior methods in perceptual and probabilistic forecasting metrics. We propose an autoregressive, end-to-end optimized video diffusion model inspired by recent advances in neural video compression. The model successively generates future frames by correcting a deterministic next-frame prediction using a stochastic residual generated by an inverse diffusion process. We compare this approach against five baselines on four datasets involving natural and simulation-based videos. We find significant improvements in terms of perceptual quality for all datasets. Furthermore, by introducing a scalable version of the Continuous Ranked Probability Score (CRPS) applicable to video, we show that our model also outperforms existing approaches in their probabilistic frame forecasting ability.

Key facts. Cited by 237 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 76 of its 100 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

76 of 100 references.

References · 76

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2013Auto-Encoding Variational Bayescited by 3,980
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
38 more
  1. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  2. 2018World Modelscited by 1,444
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  4. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  5. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  6. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  7. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  8. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  9. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  10. 2017The Perception-Distortion Tradeoffcited by 582
  11. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  12. 2018Video-to-Video Synthesiscited by 388
  13. 2016Generating Videos with Scene Dynamicscited by 376
  14. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  15. 2021Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecastingcited by 332
  16. 2018DVC: An End-to-end Deep Video Compression Frameworkcited by 324
  17. 2021Skillful Precipitation Nowcasting using Deep Generative Models of Radarcited by 308
  18. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  19. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  20. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  21. 2018Stochastic Video Generation with a Learned Priorcited by 244
  22. 2017Advances in Variational Inferencecited by 236
  23. 2017Stochastic Variational Video Predictioncited by 234
  24. 2017Video Frame Synthesis using Deep Voxel Flowcited by 209
  25. 2020A Review on Generative Adversarial Networks: Algorithms, Theory, and Applicationscited by 209
  26. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  27. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  28. 2018Stochastic Adversarial Video Predictioncited by 173
  29. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  30. 2015Delving Deeper into Convolutional Networks for Learning Video Representationscited by 146
  31. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  32. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  33. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  34. 2018Disentangled Sequential Autoencodercited by 117
  35. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  36. 2020Stochastic Latent Residual Video Predictioncited by 106
  37. 2020Latent Video Transformercited by 106
  38. 2019Improved Conditional VRNNs for Video Predictioncited by 104

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 31; 2023: 87; 2024: 76; 2025: 33; 2026: 10.

237 papers.

Sort:

By area:

  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  6. 2022Diffusion Models in Vision: A Surveycited by 1,008
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  8. 2023ModelScope Text-to-Video Technical Reportcited by 651
  9. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  10. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  11. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
38 more
  1. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  3. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  4. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  5. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  6. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  7. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  8. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  9. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  10. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  11. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  12. 2022A Survey on Generative Diffusion Modelcited by 238
  13. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 210
  14. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
  15. 2023A Survey on Video Diffusion Modelscited by 203
  16. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  17. 2022Diffusion-based Time Series Imputation and Forecasting with Structured State Space Modelscited by 195
  18. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  19. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  20. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  21. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  22. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  23. 2023GenCast: Diffusion-based ensemble forecasting for medium-range weathercited by 169
  24. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  25. 2022Lossy Image Compression with Conditional Diffusion Modelscited by 165
  26. 2022Generative Modelling With Inverse Heat Dissipationcited by 164
  27. 2023Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examplescited by 163
  28. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  29. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  30. 2022GENIE: Higher-Order Denoising Diffusion Solverscited by 127
  31. 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 119
  32. 2023DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecastingcited by 105
  33. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  34. 2024Rolling Diffusion Modelscited by 102
  35. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  36. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  37. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 90
  38. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81

50 most cited of 237 All 237 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Progressive Autoregressive Video Diffusion Models
  2. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  3. 2026Efficient Video Diffusion Models: Advancements and Challenges
  4. 2025AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset
  5. 2024Accelerating Video Diffusion Models via Distribution Matching
  6. 2024Video Diffusion Models: A Survey
  7. 2026Spectral Progressive Diffusion for Efficient Image and Video Generation
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets