mlsanityData to 7 Oct 2026

All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds

arXiv 2411.18810 ↗First posted 27 Nov 2024cs.CV cs.LGcited by 31

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image diffusion models have demonstrated remarkable capability in generating realistic images from arbitrary text prompts. However, they often produce inconsistent results for compositional prompts such as "two dogs" or "a penguin on the right of a bowl". Understanding these inconsistencies is crucial for reliable image generation. In this paper, we highlight the significant role of initial noise in these inconsistencies, where certain noise patterns are more reliable for compositional prompts than others. Our analyses reveal that different initial random seeds tend to guide the model to place objects in distinct image areas, potentially adhering to specific patterns of camera angles and image composition associated with the seed. To improve the model's compositional ability, we propose a method for mining these reliable cases, resulting in a curated training set of generated images without requiring any manual annotation. By fine-tuning text-to-image models on these generated images, we significantly enhance their compositional capabilities. For numerical composition, we observe relative increases of 29.3% and 19.5% for Stable Diffusion and PixArt-α, respectively. Spatial composition sees even larger gains, with 60.7% for Stable Diffusion and 21.1% for PixArt-α.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 35 of its 41 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

35 of 41 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization6 Apr
  2. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation25 Mar
  3. 2023Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis28 Nov
  4. 2023Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following28 Nov
  5. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation25 Oct
  6. 2023SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation20 Aug
  7. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation9 Aug
  8. 2023Zero-shot spatial layout conditioning for text-to-image diffusion models23 Jun
  9. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  10. 2023Generating images of rare concepts using pre-trained diffusion models27 Apr
  11. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation30 Mar
  12. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidance25 Feb
4 more
  1. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  2. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  7. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  8. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  11. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  12. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
7 more
  1. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  2. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  3. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  4. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  5. 2023Exploiting the Signal-Leak Bias in Diffusion Modelscited by 36
  6. 2023Generating Features with Increased Crop-related Diversity for Few-Shot Object Detectioncited by 21
  7. 2024Assessing Sample Quality via the Latent Space of Generative Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 14.

31 papers.

Sort:

By area:

  1. 2025PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Frameworkcited by 37
  2. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  3. 2025Emergence and Evolution of Interpretable Concepts in Diffusion Modelscited by 20
  4. 2025GARF: Learning Generalizable 3D Reassembly for Real-World Fracturescited by 12
  5. 2025Noise Matters: Optimizing Matching Noise for Diffusion Classifierscited by 7
  6. 2025GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Designcited by 5
  7. 2025Mitigating Diffusion Model Hallucinations with Dynamic Guidancecited by 5
  8. 2025AgentComp: From Agentic Reasoning to Compositional Mastery in Text-to-Image Modelscited by 5
  9. 2025Fine-Grained Alignment and Noise Refinement for Compositional Text-to-Image Generationcited by 4
  10. 2025When Memory Becomes a Vulnerability: Towards Multi-turn Jailbreak Attacks against Text-to-Image Generation Systemscited by 4
  11. 2025SONIC: Spectral Optimization of Noise for Inpainting with Consistencycited by 4
  12. 2025FastInit: Fast Noise Initialization for Temporally Consistent Video Generationcited by 2
19 more
  1. 2025Noise-Level Diffusion Guidance: Well Begun is Half Donecited by 2
  2. 2025CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Explorationcited by 2
  3. 2025Demystifying Numerosity in Diffusion Models -- Limitations and Remediescited by 2
  4. 2025Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformercited by 1
  5. 2026Towards Source-Aware Object Swapping with Initial Noise Perturbationcited by 1
  6. 2026LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulationcited by 1
  7. 2026A Universal Avoidance Method for Diverse Multi-branch Generationcited by 1
  8. 2026LENS: Low-Frequency Eigen Noise Shaping for Efficient Diffusion Samplingcited by 1
  9. 2026When Does High-CFG Diffusion Inversion Fail? A Controlled Study of Prompt--Latent Interactionscited by 1
  10. 2025ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generationcited by 0
  11. 2026Rethinking Refinement: Correcting Generative Bias without Noise Injectioncited by 0
  12. 2026General and Efficient Steering of Diffusion Modelscited by 0
  13. 2026ATHENA: Adaptive Test-Time Steering for Improving Count Fidelity in Diffusion Modelscited by 0
  14. 2026Boosting Text-to-Image Diffusion Models via Core Token Attention-Based Seed Selectioncited by 0
  15. 2026The Hitchhiker's Guide to Monoculture: AI Homogenizes Syntax, Not (Necessarily) Semanticscited by 0
  16. 2026NoisEasier: Test-Time Noise Optimization for Text-to-Video Generationcited by 0
  17. 2026Panoptic Scene Program Diffusion Transformercited by 0
  18. 2026Optimizing H-Graph Hybridization for Diffusion-Guided RRTcited by 0
  19. 2026Weeding Out Bad Seeds: Initial-Noise-Robust Unlearning for Text-to-Image Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward
  2. 2024Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models
  3. 2024Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation
  4. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models
  5. 2024Generating Intermediate Representations for Compositional Text-To-Image Generation
  6. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selection
  7. 2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis
  8. 2024Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings