# FreeInit: Bridging Initialization Gap in Video Diffusion Models

arXiv: https://arxiv.org/abs/2312.07537 · HTML: https://mlsanity.com/paper/2312.07537/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 46 of its 53 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)

## Built on by (most cited first, 30 of 95)

- [AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks](https://mlsanity.com/paper/2403.14468/) (2024)
- [ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation](https://mlsanity.com/paper/2402.04324/) (2024)
- [FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention](https://mlsanity.com/paper/2407.19918/) (2024)
- [SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation](https://mlsanity.com/paper/2411.04989/) (2024)
- [FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models](https://mlsanity.com/paper/2406.16863/) (2024)
- [Animate3D: Animating Any 3D Model with Multi-view Video Diffusion](https://mlsanity.com/paper/2407.11398/) (2024)
- [Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection](https://mlsanity.com/paper/2412.10891/) (2024)
- [Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation](https://mlsanity.com/paper/2406.06890/) (2024)
- [FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editing](https://mlsanity.com/paper/2407.17850/) (2024)
- [Motion Inversion for Video Customization](https://mlsanity.com/paper/2403.20193/) (2024)
- [Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model](https://mlsanity.com/paper/2406.15735/) (2024)
- [SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Device](https://mlsanity.com/paper/2412.10494/) (2024)
- [Epipolar Geometry Improves Video Generation Models](https://mlsanity.com/paper/2510.21615/) (2025)
- [FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation](https://mlsanity.com/paper/2506.01144/) (2025)
- [Deep Diffusion Image Prior for Efficient OOD Adaptation in 3D Inverse Problems](https://mlsanity.com/paper/2407.10641/) (2024)
- [FrameBridge: Improving Image-to-Video Generation with Bridge Models](https://mlsanity.com/paper/2410.15371/) (2024)
- [ObjCtrl-2.5D: Training-free Object Control with Camera Poses](https://mlsanity.com/paper/2412.07721/) (2024)
- [Streaming Autoregressive Video Generation via Diagonal Distillation](https://mlsanity.com/paper/2603.09488/) (2026)
- [Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning](https://mlsanity.com/paper/2605.21488/) (2026)
- [Optical-Flow Guided Prompt Optimization for Coherent Video Generation](https://mlsanity.com/paper/2411.15540/) (2024)
- [FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusion](https://mlsanity.com/paper/2507.00162/) (2025)
- [FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance](https://mlsanity.com/paper/2408.08189/) (2024)
- [TVG: A Training-free Transition Video Generation Method with Diffusion Models](https://mlsanity.com/paper/2408.13413/) (2024)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [Self-Refining Video Sampling](https://mlsanity.com/paper/2601.18577/) (2026)
- [UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control](https://mlsanity.com/paper/2403.02332/) (2024)
- [VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models](https://mlsanity.com/paper/2403.05438/) (2024)
- [I4VGen: Image as Free Stepping Stone for Text-to-Video Generation](https://mlsanity.com/paper/2406.02230/) (2024)
- [Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models](https://mlsanity.com/paper/2407.15642/) (2024)
- [ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videos](https://mlsanity.com/paper/2503.16400/) (2025)

## Abstract (arXiv)

Though diffusion-based video generation has witnessed rapid progress, the inference results of existing models still exhibit unsatisfactory temporal consistency and unnatural dynamics. In this paper, we delve deep into the noise initialization of video diffusion models, and discover an implicit training-inference gap that attributes to the unsatisfactory inference quality.Our key findings are: 1) the spatial-temporal frequency distribution of the initial noise at inference is intrinsically different from that for training, and 2) the denoising process is significantly influenced by the low-frequency components of the initial noise. Motivated by these observations, we propose a concise yet effective inference sampling strategy, FreeInit, which significantly improves temporal consistency of videos generated by diffusion models. Through iteratively refining the spatial-temporal low-frequency components of the initial latent during inference, FreeInit is able to compensate the initialization gap between training and inference, thus effectively improving the subject appearance and temporal consistency of generation results. Extensive experiments demonstrate that FreeInit consistently enhances the generation quality of various text-to-video diffusion models without additional training or fine-tuning.