mlsanityData to 7 Oct 2026

Diffusion Models for Video Prediction and Infilling

arXiv 2206.07696 ↗First posted 15 Jun 2022cs.CV cs.LG stat.MLcited by 134

On the map: Diffusion models → Video generation

Abstract (arXiv)

Predicting and anticipating future outcomes or reasoning about missing information in a sequence are critical skills for agents to be able to make intelligent decisions. This requires strong, temporally coherent generative capabilities. Diffusion models have shown remarkable success in several generative tasks, but have not been extensively explored in the video domain. We present Random-Mask Video Diffusion (RaMViD), which extends image diffusion models to videos using 3D convolutions, and introduces a new conditioning technique during training. By varying the mask we condition on, the model is able to perform video prediction, infilling, and upsampling. Due to our simple conditioning scheme, we can utilize the same architecture as used for unconditional training, which allows us to train the model in a conditional and unconditional fashion at the same time. We evaluate RaMViD on two benchmark datasets for video prediction, on which we achieve state-of-the-art results, and one for video generation. High-resolution videos are provided at https://sites.google.com/view/video-diffusion-prediction.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 53 of its 66 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

53 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Flexible Diffusion Modeling of Long Videos23 May
  2. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  3. 2022Video Diffusion Models7 Apr
  4. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 49

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  8. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  9. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  12. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
37 more
  1. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  2. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  3. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  4. 2016Generating Videos with Scene Dynamicscited by 376
  5. 2018A Short Note about Kinetics-600cited by 331
  6. 2017Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolationcited by 321
  7. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  8. 2020Permutation Invariant Graph Generation via Score-Based Generative Modelingcited by 263
  9. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  10. 2020Learning Gradient Fields for Shape Generationcited by 245
  11. 2018Stochastic Video Generation with a Learned Priorcited by 244
  12. 2017Stochastic Variational Video Predictioncited by 234
  13. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 230
  14. 2018Human Action Recognition and Prediction: A Surveycited by 226
  15. 2021Conditional Image Generation with Score-Based Diffusion Modelscited by 195
  16. 2019Scaling Autoregressive Video Modelscited by 189
  17. 2018Stochastic Adversarial Video Predictioncited by 173
  18. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  19. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  20. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  21. 2020Latent Video Transformercited by 106
  22. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  23. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  24. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  25. 2020Transformation-based Adversarial Video Prediction on Large-Scale Datacited by 66
  26. 2021Diffusion-Based Representation Learningcited by 57
  27. 2020Probabilistic Future Prediction for Video Scene Understandingcited by 45
  28. 2021Clockwork Variational Autoencoderscited by 40
  29. 2019Lower Dimensional Kernels for Video Discriminatorscited by 37
  30. 2015Dense Optical Flow Prediction from a Static Imagecited by 36
  31. 2018Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANscited by 36
  32. 2018Deep Energy Estimator Networkscited by 34
  33. 2019Adversarial Video Generation on Complex Datasetscited by 33
  34. 2018Recurrent Flow-Guided Semantic Forecastingcited by 19
  35. 2020A Review on Object Pose Recovery: from 3D Bounding Box Detectors to Full 6D Pose Estimatorscited by 18
  36. 2017Visual Forecasting by Imitating Dynamics in Natural Sequencescited by 14
  37. 2018Stochastic Dynamics for Video Infillingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 39; 2024: 45; 2025: 23; 2026: 15.

134 papers.

Sort:

By area:

  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  2. 2022Diffusion Models in Vision: A Surveycited by 1,008
  3. 2024Genie: Generative Interactive Environmentscited by 706
  4. 2023ModelScope Text-to-Video Technical Reportcited by 651
  5. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  6. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  7. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  8. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  9. 2022MAGVIT: Masked Generative Video Transformercited by 341
  10. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  11. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  12. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
38 more
  1. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  2. 2022A Survey on Generative Diffusion Modelcited by 238
  3. 2023A Survey on Video Diffusion Modelscited by 203
  4. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  5. 2023Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Modelscited by 172
  6. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  7. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  8. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  9. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  10. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  11. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  12. 2023Generative Image Dynamicscited by 93
  13. 2023Diffusion Action Segmentationcited by 74
  14. 2024Diffusion Models in Low-Level Vision: A Surveycited by 73
  15. 2024Vlogger: Make Your Dream A Vlogcited by 70
  16. 2022Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidancecited by 67
  17. 2023Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulationcited by 65
  18. 2022SinFusion: Training Diffusion Models on a Single Image or Videocited by 58
  19. 2023DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcastingcited by 57
  20. 2024On conditional diffusion models for PDE simulationscited by 56
  21. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 54
  22. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 54
  23. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  24. 2023Seer: Language Instructed Video Prediction with Latent Diffusion Modelscited by 51
  25. 2022RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Modelscited by 45
  26. 2022Temporally Consistent Transformers for Video Generationcited by 43
  27. 2024Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditionscited by 42
  28. 2023CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulationcited by 41
  29. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 41
  30. 2024MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generationcited by 41
  31. 2024A Survey on Long Video Generation: Challenges, Methods, and Prospectscited by 40
  32. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  33. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  34. 2023Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMscited by 35
  35. 2022Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generationcited by 32
  36. 2023Blackout Diffusion: Generative Diffusion Models in Discrete-State Spacescited by 29
  37. 2024TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Modelscited by 25
  38. 2024GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesiscited by 25

50 most cited of 134 All 134 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  2. 2023Masked Diffusion Models Are Fast Distribution Learners
  3. 2024Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model
  4. 2023Diffusion Models as Masked Autoencoders
  5. 2025TPDiff: Temporal Pyramid Video Diffusion Model
  6. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scale
  7. 2025USV: Unified Sparsification for Accelerating Video Diffusion Models
  8. 2024Accelerating Video Diffusion Models via Distribution Matching