mlsanityData to 14 Oct 2026

A Noise is Worth Diffusion Guidance

arXiv 2412.03895 ↗First posted 5 Dec 2024cs.CV cs.AI cs.LGcited by 49

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models excel in generating high-quality images. However, current diffusion models struggle to produce reliable images without guidance methods, such as classifier-free guidance (CFG). Are guidance methods truly necessary? Observing that noise obtained via diffusion inversion can reconstruct high-quality images without guidance, we focus on the initial noise of the denoising pipeline. By mapping Gaussian noise to `guidance-free noise', we uncover that small low-magnitude low-frequency components significantly enhance the denoising process, removing the need for guidance and thus improving both inference throughput and memory. Expanding on this, we propose ours, a novel method that replaces guidance methods with a single refinement of the initial noise. This refined noise enables high-quality image generation without guidance, within the same diffusion pipeline. Our noise-refining model leverages efficient noise-space learning, achieving rapid convergence and strong performance with just 50K text-image pairs. We validate its effectiveness across diverse metrics and analyze how refined noise can eliminate the need for guidance. See our project page: https://cvlab-kaist.github.io/NoiseRefine/.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 24 of its 52 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

24 of 52 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention1 Aug
  2. 2024Not All Noises Are Created Equally:Diffusion Noise Selection and Optimization19 Jul
  3. 2024No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models2 Jul
  4. 2024CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models12 Jun
  5. 2024Guiding a Diffusion Model with a Bad Version of Itself4 Jun
  6. 2024Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance26 Mar
  7. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillation21 Feb
  8. 2023CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling26 Oct
  9. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis30 Sep
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  11. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  12. 2022Classifier-Free Diffusion Guidance26 Jul
2 more
  1. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  2. 2020Denoising Diffusion Implicit Models6 Oct

Other references · 10

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  3. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  4. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
  5. 2024ReNoise: Real Image Inversion Through Iterative Noisingcited by 128
  6. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
  7. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Modelscited by 48
  8. 2023Lightning-Fast Image Inversion and Editing for Text-to-Image Diffusion Modelscited by 35
  9. 2023The Lottery Ticket Hypothesis in Denoising: Towards Semantic-Driven Initializationcited by 23
  10. 2024Model-Agnostic Human Preference Inversion in Diffusion Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 26; 2026: 22.

49 papers.

Sort:

By area:

  1. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 234
  2. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  3. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 56
  4. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 41
  5. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 40
  6. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  7. 2025Test-Time Scaling of Diffusion Models via Noise Trajectory Searchcited by 30
  8. 2026Action-to-Action Flow Matchingcited by 16
  9. 2024The Silent Assistant: NoiseQuery as Implicit Guidance for Goal-Driven Image Generationcited by 14
  10. 2025Diverse Text-to-Image Generation via Contrastive Noise Optimizationcited by 12
  11. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Modelscited by 11
  12. 2026Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matchingcited by 9
37 more
  1. 2025REG: Rectified Gradient Guidance for Conditional Diffusion Modelscited by 8
  2. 2025Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinementcited by 8
  3. 2025Towards Self-Improvement of Diffusion Models via Group Preference Optimizationcited by 7
  4. 2025RewardSDS: Aligning Score Distillation via Reward-Weighted Samplingcited by 4
  5. 2025SONIC: Spectral Optimization of Noise for Inpainting with Consistencycited by 4
  6. 2025AutoRefiner: Improving Autoregressive Video Diffusion Models via Reflective Refinement Over the Stochastic Sampling Pathcited by 4
  7. 2026HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Modelscited by 4
  8. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  9. 2026FASTER: Value-Guided Sampling for Fast RLcited by 4
  10. 2025FastInit: Fast Noise Initialization for Temporally Consistent Video Generationcited by 3
  11. 2025CoRe^2: Collect, Reflect and Refine to Generate Better and Fastercited by 2
  12. 2025TAIGen: Training-Free Adversarial Image Generation via Diffusion Modelscited by 2
  13. 2025Noise-Level Diffusion Guidance: Well Begun is Half Donecited by 2
  14. 2025UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptationcited by 2
  15. 2025TAUE: Training-free Noise Transplant and Cultivation Diffusion Modelcited by 2
  16. 2026Oracle Noise: Faster Semantic Spherical Alignment for Interpretable Latent Optimizationcited by 2
  17. 2026Colored Noise Diffusion Samplingcited by 2
  18. 2026Where Should Action Generation Begin? A Learnable Source Prior for Generative Robot Policiescited by 2
  19. 2025Foresight Diffusion: Improving Sampling Consistency in Predictive Diffusion Modelscited by 1
  20. 2025Brownian Bridge Diffusion for Sequential Recommendationcited by 1
  21. 2025Inference-Time Scaling of Diffusion Models for Infrared Data Generationcited by 1
  22. 2025Rethinking Test Time Scaling for Flow-Matching Generative Modelscited by 1
  23. 2025Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generationcited by 1
  24. 2026Towards Source-Aware Object Swapping with Initial Noise Perturbationcited by 1
  25. 2026LENS: Low-Frequency Eigen Noise Shaping for Efficient Diffusion Samplingcited by 1
  26. 2026NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Trainingcited by 1
  27. 2026StressDream: Steering Video World Models for Robust Policy Evaluation and Improvementcited by 1
  28. 2026Self-Improving VLA Policies: Selected Diffusion Noise for Spurious-Robust Action Smoothingcited by 1
  29. 2026ReconPlusGen: Injecting Reconstruction Prior into Multi-view 3D Generation through Noise Inversion and Modulationcited by 1
  30. 2026APPLE: Attribute-Preserving Pseudo-Labeling for Diffusion-Based Face Swappingcited by 0
  31. 2026Does Semantic Noise Initialization Transfer from Images to Videos? A Paired Diagnostic Studycited by 0
  32. 2026Parallel Tempering Initial Sampling in Inference-Time Reward Alignmentcited by 0
  33. 2026Adaptive Inference-Time Scaling via Early-Step Latent Verification for Image Editingcited by 0
  34. 2026Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Timecited by 0
  35. 2026TISC: A Text-Driven Image Semantic Communication System for Faithful Reconstructioncited by 0
  36. 2026Unlocking Few-Step Diffusion for Faithful Previewscited by 0
  37. 2026Correction Training for Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Selective Guidance: Are All the Denoising Steps of Guided Diffusion Important?
  2. 2025GuidNoise: Single-Pair Guided Diffusion for Generalized Noise Synthesis
  3. 2025Mitigating the Noise Shift for Denoising Generative Models via Noise Awareness Guidance
  4. 2024There and Back Again: On the relation between Noise and Image Inversions in Diffusion Models
  5. 2023Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models
  6. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidance
  7. 2025Navigating with Annealing Guidance Scale in Diffusion Space
  8. 2025Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorations