mlsanityData to 7 Oct 2026

Flexible Diffusion Modeling of Long Videos

arXiv 2205.11495 ↗First posted 23 May 2022cs.CV cs.LGcited by 294

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames. We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA autonomous driving simulator.

Key facts. Cited by 294 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 36 of its 44 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

36 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Video Diffusion Models7 Apr
  2. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
22 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  3. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  4. 2019Model-Based Reinforcement Learning for Ataricited by 495
  5. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  6. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  7. 2018Stochastic Video Generation with a Learned Priorcited by 244
  8. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  9. 2017Stochastic Variational Video Predictioncited by 234
  10. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  11. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  12. 2019Scaling Autoregressive Video Modelscited by 189
  13. 2021Symbolic Music Generation with Diffusion Modelscited by 152
  14. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
  15. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  16. 2016Memory-Efficient Backpropagation Through Timecited by 78
  17. 2020Solving Linear Inverse Problems Using the Prior Implicit in a Denoisercited by 78
  18. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  19. 2021Clockwork Variational Autoencoderscited by 40
  20. 2017Unbiasing Truncated Backpropagation Through Timecited by 37
  21. 2019Variational Temporal Abstractioncited by 37
  22. 2020Video Generative Adversarial Networks: A Reviewcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 90; 2024: 103; 2025: 60; 2026: 21.

294 papers.

Sort:

By area:

  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  6. 2022Diffusion Models in Vision: A Surveycited by 1,008
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  8. 2023ModelScope Text-to-Video Technical Reportcited by 651
  9. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  10. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  11. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  12. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
38 more
  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  3. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  4. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  6. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  7. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  8. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  9. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  10. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  11. 2022EDGE: Editable Dance Generation From Musiccited by 284
  12. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  13. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  14. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  15. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  16. 2022A Survey on Generative Diffusion Modelcited by 238
  17. 2023A Survey on Video Diffusion Modelscited by 203
  18. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  19. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  20. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  21. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  22. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
  23. 2023Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Modelscited by 172
  24. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  25. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  26. 2023GenCast: Diffusion-based ensemble forecasting for medium-range weathercited by 169
  27. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  28. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  29. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  30. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  31. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  32. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  33. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 134
  34. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 126
  35. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  36. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  37. 2023DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecastingcited by 105
  38. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103

50 most cited of 294 All 294 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Progressive Autoregressive Video Diffusion Models
  2. 2025VideoMerge: Towards Training-free Long Video Generation
  3. 2026Efficient Video Diffusion Models: Advancements and Challenges
  4. 2025AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset
  5. 2026Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model
  6. 2025USV: Unified Sparsification for Accelerating Video Diffusion Models
  7. 2026Test-Time Scaling for Video Diffusion Models via Diagnosis-Guided Candidate Recycling
  8. 2026FlowLong: Inference-time Long Video Generation via Manifold-constrained Tweedie Matching