mlsanityData to 7 Oct 2026

Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models

arXiv 2305.10474 ↗First posted 17 May 2023cs.CV cs.GR cs.LGcited by 284

On the map: Diffusion models → Video generation

Abstract (arXiv)

Despite tremendous progress in generating high-quality images using diffusion models, synthesizing a sequence of animated frames that are both photorealistic and temporally coherent is still in its infancy. While off-the-shelf billion-scale datasets for image generation are available, collecting similar video data of the same scale is still challenging. Also, training a video diffusion model is computationally much more expensive than its image counterpart. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. We find that naively extending the image noise prior to video noise prior in video diffusion leads to sub-optimal performance. Our carefully designed video noise prior leads to substantially better performance. Extensive experimental validation shows that our model, Preserve Your Own Correlation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It also achieves SOTA video generation quality on the small-scale UCF-101 benchmark with a 10× smaller model using significantly less computation than the prior art.

Key facts. Cited by 284 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 66 of its 73 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

66 of 73 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  2. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  3. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  4. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  5. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  6. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  7. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  8. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  10. 2022SinFusion: Training Diffusion Models on a Single Image or Video21 Nov
  11. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
5 more
  1. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  2. 2022Flexible Diffusion Modeling of Long Videos23 May
  3. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  4. 2022Video Diffusion Models7 Apr
  5. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
37 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2018Group Normalizationcited by 1,514
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  9. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  10. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  11. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  12. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  13. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  14. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  15. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  16. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 526
  17. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  18. 2023Zero-shot Image-to-Image Translationcited by 503
  19. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  20. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  21. 2016Generating Videos with Scene Dynamicscited by 376
  22. 2022MAGVIT: Masked Generative Video Transformercited by 341
  23. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  24. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  25. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  26. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  27. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  28. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
  29. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  30. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  31. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  32. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  33. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
  34. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  35. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  36. 2019Lower Dimensional Kernels for Video Discriminatorscited by 37
  37. 2023MoStGAN-V: Video Generation with Temporal Motion Stylescited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 59; 2024: 133; 2025: 68; 2026: 24.

284 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  3. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  5. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  6. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  7. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  8. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  9. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  10. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  11. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  12. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
38 more
  1. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  2. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  3. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  4. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  5. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  6. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  7. 2023A Survey on Video Diffusion Modelscited by 203
  8. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  9. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
  10. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  11. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  12. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  13. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  14. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  15. 2025Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Modelscited by 160
  16. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  17. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  18. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  19. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  20. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 131
  21. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  22. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  23. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  24. 2024Training-free Camera Control for Video Generationcited by 119
  25. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  26. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
  27. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  28. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  29. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  30. 2025Long Context Tuning for Video Generationcited by 110
  31. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  32. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  33. 2024Rolling Diffusion Modelscited by 102
  34. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  35. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  36. 2023FreeInit: Bridging Initialization Gap in Video Diffusion Modelscited by 93
  37. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 91
  38. 2023VideoLCM: Video Latent Consistency Modelcited by 90

50 most cited of 284 All 284 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Video Diffusion Models: Advancements and Challenges
  2. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  3. 2024Frame by Familiar Frame: Understanding Replication in Video Diffusion Models
  4. 2024Progressive Autoregressive Video Diffusion Models
  5. 2026Spectral Progressive Diffusion for Efficient Image and Video Generation
  6. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  7. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videos
  8. 2024Noise Crystallization and Liquid Noise: Zero-shot Video Generation using Image Diffusion Models