mlsanityData to 14 Oct 2026

Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction

arXiv 2412.04929 ↗First posted 6 Dec 2024cs.CV cs.AI cs.LGcited by 1

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion models have made significant strides in image generation, mastering tasks such as unconditional image synthesis, text-image translation, and image-to-image conversions. However, their capability falls short in the realm of video prediction, mainly because they treat videos as a collection of independent images, relying on external constraints such as temporal attention mechanisms to enforce temporal coherence. In our paper, we introduce a novel model class, that treats video as a continuous multi-dimensional process rather than a series of discrete frames. We also report a reduction of 75% sampling steps required to sample a new frame thus making our framework more efficient during the inference time. Through extensive experimentation, we establish state-of-the-art performance in video prediction, validated on benchmark datasets including KTH, BAIR, Human3.6M, and UCF101. Navigate to the project page https://www.cs.umd.edu/~gauravsh/cvp/supp/website.html for video results.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 33 of its 53 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

33 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  6. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  7. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 276
  8. 2018Stochastic Video Generation with a Learned Priorcited by 245
  9. 2018Stochastic Adversarial Video Predictioncited by 173
  10. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 169
  11. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restorationcited by 165
  12. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
20 more
  1. 2020Stochastic Latent Residual Video Predictioncited by 108
  2. 2020Latent Video Transformercited by 106
  3. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  4. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  5. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  6. 2017The Pose Knows: Video Forecasting by Generating Pose Futurescited by 88
  7. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  8. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  9. 2022NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesiscited by 71
  10. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69
  11. 2020Transformation-based Adversarial Video Prediction on Large-Scale Datacited by 66
  12. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 55
  13. 2017Deep Video Generation, Prediction and Completion of Human Action Sequencescited by 52
  14. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  15. 2017Folded Recurrent Neural Networks for Future Video Predictioncited by 44
  16. 2021SLAMP: Stochastic Latent Appearance and Motion Predictioncited by 32
  17. 2021Diverse Video Generation using a Gaussian Process Triggercited by 17
  18. 2021Accurate Grid Keypoint Learning for Efficient Video Predictioncited by 13
  19. 2023Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancementscited by 5
  20. 2016Video Ladder Networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15

Similar, not cited

Close in wording, no citation either way

  1. 2022Video Diffusion Models
  2. 2024Efficient Continuous Video Flow Model for Video Prediction
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  4. 2026Efficient Video Diffusion Models: Advancements and Challenges
  5. 2024Video Diffusion Models: A Survey
  6. 2024Frame by Familiar Frame: Understanding Replication in Video Diffusion Models
  7. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
  8. 2023STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Prediction