mlsanityData to 7 Oct 2026

MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation

arXiv 2205.09853 ↗First posted 19 May 2022cs.CV cs.AI cs.LGcited by 307

On the map: Diffusion models → Video generation

Abstract (arXiv)

Video prediction is a challenging task. The quality of video frames from current state-of-the-art (SOTA) generative models tends to be poor and generalization beyond the training data is difficult. Furthermore, existing prediction frameworks are typically not capable of simultaneously handling other video-related tasks such as unconditional generation or interpolation. In this work, we devise a general-purpose framework called Masked Conditional Video Diffusion (MCVD) for all of these video synthesis tasks using a probabilistic conditional score-based denoising diffusion model, conditioned on past and/or future frames. We train the model in a manner where we randomly and independently mask all the past frames or all the future frames. This novel but straightforward setup allows us to train a single model that is capable of executing a broad range of video tasks, specifically: future/past prediction -- when only future/past frames are masked; unconditional generation -- when both past and future frames are masked; and interpolation -- when neither past nor future frames are masked. Our experiments show that this approach can generate high-quality frames for diverse types of videos. Our MCVD models are built from simple non-recurrent 2D-convolutional architectures, conditioning on blocks of frames and generating blocks of frames. We generate videos of arbitrary lengths autoregressively in a block-wise manner. Our approach yields SOTA results across standard video prediction and interpolation benchmarks, with computation times for training models measured in 1-12 days using ≤ 4 GPUs. Project page: https://mask-cond-video-diffusion.github.io ; Code : https://github.com/voletiv/mcvd-pytorch

Key facts. Cited by 307 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 70 of its 88 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

70 of 88 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Video Diffusion Models7 Apr
  2. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  3. 2016Improved Techniques for Training GANscited by 2,938
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  5. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  6. 2018Group Normalizationcited by 1,514
  7. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  8. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  9. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  10. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  11. 2022Denoising Diffusion Restoration Modelscited by 951
  12. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  13. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  14. 2017Video Enhancement with Task-Oriented Flowcited by 612
  15. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  16. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  17. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  18. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  19. 2016Generating Videos with Scene Dynamicscited by 376
  20. 2017Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolationcited by 321
  21. 2018Stochastic Video Generation with a Learned Priorcited by 244
  22. 2017Stochastic Variational Video Predictioncited by 234
  23. 2019Depth-Aware Video Frame Interpolationcited by 232
  24. 2014Efficient Object Localization Using Convolutional Networkscited by 226
  25. 2021Gotta Go Fast When Generating Data with Score-Based Modelscited by 218
  26. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  27. 2019Scaling Autoregressive Video Modelscited by 189
  28. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  29. 2018Stochastic Adversarial Video Predictioncited by 173
  30. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  31. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  32. 2017Video Frame Interpolation via Adaptive Convolutioncited by 158
  33. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  34. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  35. 2020Stochastic Latent Residual Video Predictioncited by 106
  36. 2020Latent Video Transformercited by 106
  37. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  38. 2020Adversarial score matching and improved sampling for image generationcited by 95

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 84; 2024: 106; 2025: 73; 2026: 29.

307 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  5. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  6. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  7. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  8. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  9. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  10. 2022MAGVIT: Masked Generative Video Transformercited by 341
  11. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  12. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
38 more
  1. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  2. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  3. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  4. 2024Navigation World Modelscited by 262
  5. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  6. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  7. 2022A Survey on Generative Diffusion Modelcited by 238
  8. 2023A Survey on Video Diffusion Modelscited by 203
  9. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  10. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  11. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  12. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
  13. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  14. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  15. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  16. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  17. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 143
  18. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  19. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  20. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  21. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  22. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 134
  23. 2023DiffusionSat: A Generative Foundation Model for Satellite Imagerycited by 130
  24. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 128
  25. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  26. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  27. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  28. 2023DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecastingcited by 105
  29. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  30. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  31. 2022Refining Generative Process with Discriminator Guidance in Score-based Diffusion Modelscited by 98
  32. 2023PreDiff: Precipitation Nowcasting with Latent Diffusion Modelscited by 96
  33. 2023Generative Image Dynamicscited by 93
  34. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 90
  35. 2023LDMVFI: Video Frame Interpolation with Latent Diffusion Modelscited by 88
  36. 2024FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attentioncited by 87
  37. 2024Constrained Synthesis with Projected Diffusion Modelscited by 83
  38. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83

50 most cited of 307 All 307 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model
  2. 2025Conditional Video Generation for High-Efficiency Video Compression
  3. 2026Efficient Video Diffusion Models: Advancements and Challenges
  4. 2025AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset
  5. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis
  6. 2023Masked Diffusion Models Are Fast Distribution Learners
  7. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  8. 2024Conditional Image Synthesis with Diffusion Models: A Survey