mlsanityData to 7 Oct 2026

Video Probabilistic Diffusion Models in Projected Latent Space

arXiv 2302.07685 ↗First posted 15 Feb 2023cs.CV cs.LGcited by 171

On the map: Diffusion models → Video generation

Abstract (arXiv)

Despite the remarkable progress in deep generative models, synthesizing high-resolution and temporally coherent videos still remains a challenge due to their high-dimensionality and complex temporal dynamics along with large spatial variations. Recent works on diffusion models have shown their potential to solve this challenge, yet they suffer from severe computation- and memory-inefficiency that limit the scalability. To handle this issue, we propose a novel generative model for videos, coined projected latent video diffusion models (PVDM), a probabilistic diffusion model which learns a video distribution in a low-dimensional latent space and thus can be efficiently trained with high-resolution videos under limited resources. Specifically, PVDM is composed of two components: (a) an autoencoder that projects a given video as 2D-shaped latent vectors that factorize the complex cubic structure of video pixels and (b) a diffusion model architecture specialized for our new factorized latent space and the training/sampling procedure to synthesize videos of arbitrary length with a single model. Experiments on popular video generation datasets demonstrate the superiority of PVDM compared with previous video synthesis methods; e.g., PVDM obtains the FVD score of 639.7 on the UCF-101 long video (128 frames) generation benchmark, which improves 1773.4 of the prior state-of-the-art.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 61 of its 84 references that are in the map. First posted on arXiv 15 Feb 2023.

Builds on

61 of 84 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022VIDM: Video Implicit Diffusion Models1 Dec
  2. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  4. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  5. 2022Flexible Diffusion Modeling of Long Videos23 May
  6. 2022Video Diffusion Models7 Apr
  7. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2017Neural Discrete Representation Learningcited by 4,650
38 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  10. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  11. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  12. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  13. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  14. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  15. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  16. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  17. 2021Score-based Generative Modeling in Latent Spacecited by 634
  18. 2020Jukebox: A Generative Model for Musiccited by 535
  19. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 526
  20. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  21. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  22. 2016Generating Videos with Scene Dynamicscited by 376
  23. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  24. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  25. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  26. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  27. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  28. 2019Scaling Autoregressive Video Modelscited by 189
  29. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  30. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  31. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  32. 2016Video Pixel Networkscited by 108
  33. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  34. 2020Latent Video Transformercited by 106
  35. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 88
  36. 2021StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGANcited by 47
  37. 2019Lower Dimensional Kernels for Video Discriminatorscited by 37
  38. 2018Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANscited by 36

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 51; 2024: 82; 2025: 29; 2026: 9.

171 papers.

Sort:

By area:

  1. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  2. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  3. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  4. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  5. 2023Learning Interactive Real-World Simulatorscited by 440
  6. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  7. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  8. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  9. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  10. 2023A Survey on Video Diffusion Modelscited by 203
  11. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  12. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
38 more
  1. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  2. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  3. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  4. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 134
  5. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  6. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  7. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  8. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 117
  9. 2023VidToMe: Video Token Merging for Zero-Shot Video Editingcited by 111
  10. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  11. 2024Rolling Diffusion Modelscited by 102
  12. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Modelscited by 99
  13. 2023PreDiff: Precipitation Nowcasting with Latent Diffusion Modelscited by 96
  14. 2023Generative Image Dynamicscited by 93
  15. 2023Compositional Foundation Models for Hierarchical Planningcited by 92
  16. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  17. 2023VideoLCM: Video Latent Consistency Modelcited by 90
  18. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  19. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  20. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  21. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharingcited by 70
  22. 2023SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulationcited by 66
  23. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 66
  24. 2024COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editingcited by 59
  25. 2023CDDM: Channel Denoising Diffusion Models for Wireless Semantic Communicationscited by 58
  26. 2024CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilitiescited by 58
  27. 2023DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcastingcited by 57
  28. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attentioncited by 55
  29. 2025AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusioncited by 53
  30. 2023Seer: Language Instructed Video Prediction with Latent Diffusion Modelscited by 51
  31. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesiscited by 51
  32. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51
  33. 2024On the Content Bias in Fr'echet Video Distancecited by 51
  34. 2024Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptationcited by 50
  35. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  36. 2024Improved Video VAE for Latent Video Diffusion Modelcited by 46
  37. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 41
  38. 2023LEO: Generative Latent Image Animator for Human Video Synthesiscited by 40

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
  4. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  5. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation
  6. 2025Generative Pre-trained Autoregressive Diffusion Transformer
  7. 2024Progressive Autoregressive Video Diffusion Models
  8. 2025VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate