mlsanityData to 7 Oct 2026

FreeInit: Bridging Initialization Gap in Video Diffusion Models

arXiv 2312.07537 ↗First posted 12 Dec 2023cs.CVcited by 93

On the map: Diffusion models → Video generation

Abstract (arXiv)

Though diffusion-based video generation has witnessed rapid progress, the inference results of existing models still exhibit unsatisfactory temporal consistency and unnatural dynamics. In this paper, we delve deep into the noise initialization of video diffusion models, and discover an implicit training-inference gap that attributes to the unsatisfactory inference quality.Our key findings are: 1) the spatial-temporal frequency distribution of the initial noise at inference is intrinsically different from that for training, and 2) the denoising process is significantly influenced by the low-frequency components of the initial noise. Motivated by these observations, we propose a concise yet effective inference sampling strategy, FreeInit, which significantly improves temporal consistency of videos generated by diffusion models. Through iteratively refining the spatial-temporal low-frequency components of the initial latent during inference, FreeInit is able to compensate the initialization gap between training and inference, thus effectively improving the subject appearance and temporal consistency of generation results. Extensive experiments demonstrate that FreeInit consistently enhances the generation quality of various text-to-video diffusion models without additional training or fine-tuning.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

46 of 53 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  2. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling23 Oct
  3. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  4. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  5. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation7 Sep
  6. 2023ModelScope Text-to-Video Technical Report12 Aug
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  8. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  9. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  10. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  11. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  12. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
3 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  2. 2022Flexible Diffusion Modeling of Long Videos23 May
  3. 2022Video Diffusion Models7 Apr

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
19 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  9. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  10. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  11. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  12. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  13. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  14. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  15. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  16. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  17. 2022Generating Long Videos of Dynamic Scenescited by 113
  18. 2023Text2Performer: Text-Driven Human Video Generationcited by 53
  19. 2023Exploiting the Signal-Leak Bias in Diffusion Modelscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 39; 2025: 31; 2026: 23.

93 papers.

Sort:

By area:

  1. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  2. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  3. 2024FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attentioncited by 87
  4. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  5. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 55
  6. 2024Animate3D: Animating Any 3D Model with Multi-view Video Diffusioncited by 55
  7. 2024Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillationcited by 38
  8. 2024Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflectioncited by 38
  9. 2024FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editingcited by 31
  10. 2024Motion Inversion for Video Customizationcited by 30
  11. 2024Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Modelcited by 27
  12. 2024SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Devicecited by 25
38 more
  1. 2025Epipolar Geometry Improves Video Generation Modelscited by 22
  2. 2024Deep Diffusion Image Prior for Efficient OOD Adaptation in 3D Inverse Problemscited by 21
  3. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 21
  4. 2024FrameBridge: Improving Image-to-Video Generation with Bridge Modelscited by 19
  5. 2024ObjCtrl-2.5D: Training-free Object Control with Camera Posescited by 18
  6. 2024Optical-Flow Guided Prompt Optimization for Coherent Video Generationcited by 17
  7. 2024FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidancecited by 15
  8. 2024TVG: A Training-free Transition Video Generation Method with Diffusion Modelscited by 15
  9. 2025FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusioncited by 15
  10. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  11. 2026Streaming Autoregressive Video Generation via Diagonal Distillationcited by 15
  12. 2024UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Controlcited by 14
  13. 2024Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Modelscited by 14
  14. 2026Self-Refining Video Samplingcited by 14
  15. 2024VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Modelscited by 13
  16. 2024I4VGen: Image as Free Stepping Stone for Text-to-Video Generationcited by 12
  17. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videoscited by 12
  18. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  19. 2025Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Modelscited by 11
  20. 2025Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusioncited by 10
  21. 2025Model Already Knows the Best Noise: Bayesian Active Noise Selection via Attention in Video Diffusion Modelcited by 10
  22. 20253D-LATTE: Latent Space 3D Editing from Textual Instructionscited by 10
  23. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Modelscited by 9
  24. 2024IV-Mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesiscited by 8
  25. 2025FlowDrag: 3D-aware Drag-based Image Editing with Mesh-guided Deformation Vector Flow Fieldscited by 8
  26. 2024SCP-Diff: Spatial-Categorical Joint Prior for Diffusion Based Semantic Image Synthesiscited by 7
  27. 2024ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Waycited by 7
  28. 2024SMITE: Segment Me In TimEcited by 7
  29. 2025Textured 3D Regenerative Morphing with 3D Diffusion Priorcited by 6
  30. 2024AudioScenic: Audio-Driven Video Scene Editingcited by 5
  31. 2024CoNo: Consistency Noise Injection for Tuning-free Long Video Diffusioncited by 5
  32. 2024VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guidecited by 5
  33. 2024HANDI: Hand-Centric Text-and-Image Conditioned Video Generationcited by 5
  34. 2025Point Prompting: Counterfactual Tracking with Video Diffusion Modelscited by 5
  35. 2024RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidancecited by 4
  36. 2024MAVIN: Multi-Action Video Generation with Diffusion Models via Transition Video Infillingcited by 4
  37. 2025RAGME: Retrieval Augmented Video Generation for Enhanced Motion Realismcited by 4
  38. 2025Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidancecited by 4

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Training-free Diffusion Acceleration with Bottleneck Sampling
  2. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Training
  3. 2025AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion
  4. 2025FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  6. 2024Progressive Autoregressive Video Diffusion Models
  7. 2026Test-Time Scaling for Video Diffusion Models via Diagnosis-Guided Candidate Recycling
  8. 2024Frame by Familiar Frame: Understanding Replication in Video Diffusion Models