mlsanityData to 7 Oct 2026

InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization

arXiv 2404.04650 ↗First posted 6 Apr 2024cs.CVcited by 123

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent strides in the development of diffusion models, exemplified by advancements such as Stable Diffusion, have underscored their remarkable prowess in generating visually compelling images. However, the imperative of achieving a seamless alignment between the generated image and the provided prompt persists as a formidable challenge. This paper traces the root of these difficulties to invalid initial noise, and proposes a solution in the form of Initial Noise Optimization (InitNO), a paradigm that refines this noise. Considering text prompts, not all random noises are effective in synthesizing semantically-faithful images. We design the cross-attention response score and the self-attention conflict score to evaluate the initial noise, bifurcating the initial latent space into valid and invalid sectors. A strategically crafted noise optimization pipeline is developed to guide the initial noise towards valid regions. Our method, validated through rigorous experimentation, shows a commendable proficiency in generating images in strict accordance with text prompts. Our code is available at https://github.com/xiefan-guo/initno.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 42 of its 45 references that are in the map. First posted on arXiv 6 Apr 2024.

Builds on

42 of 45 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation25 Oct
  2. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursing20 Jul
  3. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  4. 2023A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesis26 Jun
  5. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths29 May
  6. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  7. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  9. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
19 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  3. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  4. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  5. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  6. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  7. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  8. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  9. 2022On Distillation of Guided Diffusion Modelscited by 721
  10. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  11. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  12. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  13. 2023Zero-shot Image-to-Image Translationcited by 503
  14. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  15. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  16. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  17. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  18. 2023Matryoshka Diffusion Modelscited by 70
  19. 2023DreaMoving: A Human Video Generation Framework based on Diffusion Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 57; 2026: 47.

123 papers.

Sort:

By area:

  1. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  2. 2024Golden Noise for Diffusion Models: A Learning Frameworkcited by 91
  3. 2024Not All Noises Are Created Equally:Diffusion Noise Selection and Optimizationcited by 70
  4. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Modelscited by 47
  5. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
  6. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 37
  7. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  8. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  9. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  10. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seedscited by 31
  11. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  12. 2024DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Schedulingcited by 27
38 more
  1. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  2. 2024The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noisecited by 18
  3. 2024Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjectscited by 16
  4. 2024The Silent Assistant: NoiseQuery as Implicit Guidance for Goal-Driven Image Generationcited by 14
  5. 2025Towards Transformer-Based Aligned Generation with Self-Coherence Guidancecited by 14
  6. 2024I4VGen: Image as Free Stepping Stone for Text-to-Video Generationcited by 12
  7. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videoscited by 12
  8. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  9. 2024TKG-DM: Training-free Chroma Key Content Generation Diffusion Modelcited by 11
  10. 2024Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generationcited by 11
  11. 2025Physics-Informed Distillation of Diffusion Models for PDE-Constrained Generationcited by 11
  12. 2025Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesiscited by 10
  13. 2025ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generationcited by 10
  14. 2025Model Already Knows the Best Noise: Bayesian Active Noise Selection via Attention in Video Diffusion Modelcited by 10
  15. 2025Be Decisive: Noise-Induced Layouts for Multi-Subject Generationcited by 10
  16. 2025YOLO-Count: Differentiable Object Counting for Text-to-Image Generationcited by 10
  17. 2025Diverse Text-to-Image Generation via Contrastive Noise Optimizationcited by 10
  18. 2024Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesiscited by 9
  19. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Modelscited by 9
  20. 2025NoiseAR: AutoRegressing Initial Noise Prior for Diffusion Modelscited by 8
  21. 2025PINO: Person-Interaction Noise Optimization for Long-Duration and Customizable Motion Generation of Arbitrary-Sized Groupscited by 8
  22. 2024CoCoNO: Attention Contrast-and-Complete for Initial Noise Optimization in Text-to-Image Synthesiscited by 7
  23. 2025Noise Matters: Optimizing Matching Noise for Diffusion Classifierscited by 7
  24. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 7
  25. 2025Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Modelscited by 7
  26. 2024Adaptive Diffusion Terrain Generator for Autonomous Uneven Terrain Navigationcited by 6
  27. 2025Towards a Golden Classifier-Free Guidance Path via Foresight Fixed Point Iterationscited by 6
  28. 2026Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generationcited by 6
  29. 2025Early Timestep Zero-Shot Candidate Selection for Instruction-Guided Image Editingcited by 5
  30. 2025ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuningcited by 5
  31. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 5
  32. 2026Layer-wise Instance Binding for Regional and Occlusion Control in Text-to-Image Diffusion Transformerscited by 5
  33. 2024Detection-Driven Object Count Optimization for Text-to-Image Diffusion Modelscited by 4
  34. 2025Fine-Grained Alignment and Noise Refinement for Compositional Text-to-Image Generationcited by 4
  35. 2025Antithetic Noise in Diffusion Modelscited by 4
  36. 2025Steer Away From Mode Collisions: Improving Composition In Diffusion Modelscited by 4
  37. 2025SONIC: Spectral Optimization of Noise for Inpainting with Consistencycited by 4
  38. 2025Guiding What Not to Generate: Automated Negative Prompting for Text-Image Alignmentcited by 4

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Early Quality Assessment of Text-to-Image Diffusion Models
  2. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  3. 2025A Minimalist Method for Fine-tuning Text-to-Image Diffusion Models
  4. 2026Does Semantic Noise Initialization Transfer from Images to Videos? A Paired Diagnostic Study
  5. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  6. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
  7. 2026ImageRAGTurbo: Towards One-step Text-to-Image Generation with Retrieval-Augmented Diffusion Models
  8. 2024Spatial-Aware Latent Initialization for Controllable Image Generation