mlsanityData to 7 Oct 2026

VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation

arXiv 2303.08320 ↗First posted 15 Mar 2023cs.CVcited by 330

On the map: Diffusion models → Video generation

Abstract (arXiv)

A diffusion probabilistic model (DPM), which constructs a forward diffusion process by gradually adding noise to data points and learns the reverse denoising process to generate new samples, has been shown to handle complex data distribution. Despite its recent success in image synthesis, applying DPMs to video generation is still challenging due to high-dimensional data spaces. Previous methods usually adopt a standard diffusion process, where frames in the same video clip are destroyed with independent noises, ignoring the content redundancy and temporal correlation. This work presents a decomposed diffusion process via resolving the per-frame noise into a base noise that is shared among all frames and a residual noise that varies along the time axis. The denoising pipeline employs two jointly-learned networks to match the noise decomposition accordingly. Experiments on various datasets confirm that our approach, termed as VideoFusion, surpasses both GAN-based and diffusion-based alternatives in high-quality video generation. We further show that our decomposed formulation can benefit from pre-trained image diffusion models and well-support text-conditioned video creation.

Key facts. Cited by 330 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

46 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Flexible Diffusion Modeling of Long Videos23 May
  2. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  3. 2022Video Diffusion Models7 Apr
  4. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2017Neural Discrete Representation Learningcited by 4,650
30 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  10. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  11. 2020First Order Motion Model for Image Animationcited by 698
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  13. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  14. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  15. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  16. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  17. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  18. 2016Generating Videos with Scene Dynamicscited by 376
  19. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  20. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  21. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  22. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  23. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  24. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
  25. 2021UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Modelscited by 133
  26. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  27. 2022Generating Long Videos of Dynamic Scenescited by 113
  28. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 88
  29. 2020Hierarchical Autoregressive Modeling for Neural Video Compressioncited by 28
  30. 2022From Points to Functions: Infinite-dimensional Representations in Diffusion Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 94; 2024: 128; 2025: 78; 2026: 30.

330 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  3. 2023ModelScope Text-to-Video Technical Reportcited by 651
  4. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  5. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  6. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  7. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  8. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  9. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  10. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  11. 2023DeepCache: Accelerating Diffusion Models for Freecited by 346
  12. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
38 more
  1. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  2. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  3. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  4. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  5. 2023A Survey on Video Diffusion Modelscited by 203
  6. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  7. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  8. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  9. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  10. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
  11. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  12. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformerscited by 169
  13. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  14. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  15. 2024L4GM: Large 4D Gaussian Reconstruction Modelcited by 143
  16. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  17. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  18. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  19. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  20. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  21. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 134
  22. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 131
  23. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  24. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  25. 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 119
  26. 2024Training-free Camera Control for Video Generationcited by 119
  27. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  28. 2023Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusioncited by 114
  29. 2023Animate124: Animating One Image to 4D Dynamic Scenecited by 113
  30. 2025Long Context Tuning for Video Generationcited by 110
  31. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  32. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  33. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  34. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  35. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  36. 2023PreDiff: Precipitation Nowcasting with Latent Diffusion Modelscited by 96
  37. 2023Generative Image Dynamicscited by 93
  38. 2023FreeInit: Bridging Initialization Gap in Video Diffusion Modelscited by 93

50 most cited of 330 All 330 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Accelerating Video Diffusion Models via Distribution Matching
  2. 2022SinFusion: Training Diffusion Models on a Single Image or Video
  3. 2024Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generation
  4. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  5. 2025AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset
  6. 2024Decoupled Video Generation with Chain of Training-free Diffusion Model Experts
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  8. 2024SF-V: Single Forward Video Generation Model