# All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds

arXiv: https://arxiv.org/abs/2411.18810 · HTML: https://mlsanity.com/paper/2411.18810/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 35 of its 41 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://mlsanity.com/paper/2305.08891/) (2023)
- [Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack](https://mlsanity.com/paper/2309.15807/) (2023)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models](https://mlsanity.com/paper/2305.13655/) (2023)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis](https://mlsanity.com/paper/2008.05865/) (2020)
- [InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization](https://mlsanity.com/paper/2404.04650/) (2024)
- [LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation](https://mlsanity.com/paper/2308.05095/) (2023)
- [Generating images of rare concepts using pre-trained diffusion models](https://mlsanity.com/paper/2304.14530/) (2023)
- [Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following](https://mlsanity.com/paper/2311.17002/) (2023)
- [Directed Diffusion: Direct Control of Object Placement through Attention Guidance](https://mlsanity.com/paper/2302.13153/) (2023)
- [Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation](https://mlsanity.com/paper/2403.16990/) (2024)
- [Zero-shot spatial layout conditioning for text-to-image diffusion models](https://mlsanity.com/paper/2306.13754/) (2023)
- [A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation](https://mlsanity.com/paper/2310.16656/) (2023)

## Built on by (most cited first, 30 of 31)

- [PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Framework](https://mlsanity.com/paper/2506.10741/) (2025)
- [IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout](https://mlsanity.com/paper/2506.01949/) (2025)
- [Emergence and Evolution of Interpretable Concepts in Diffusion Models](https://mlsanity.com/paper/2504.15473/) (2025)
- [GARF: Learning Generalizable 3D Reassembly for Real-World Fractures](https://mlsanity.com/paper/2504.05400/) (2025)
- [Noise Matters: Optimizing Matching Noise for Diffusion Classifiers](https://mlsanity.com/paper/2508.11330/) (2025)
- [When Memory Becomes a Vulnerability: Towards Multi-turn Jailbreak Attacks against Text-to-Image Generation Systems](https://mlsanity.com/paper/2504.20376/) (2025)
- [GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design](https://mlsanity.com/paper/2508.15227/) (2025)
- [Mitigating Diffusion Model Hallucinations with Dynamic Guidance](https://mlsanity.com/paper/2510.05356/) (2025)
- [AgentComp: From Agentic Reasoning to Compositional Mastery in Text-to-Image Models](https://mlsanity.com/paper/2512.09081/) (2025)
- [Fine-Grained Alignment and Noise Refinement for Compositional Text-to-Image Generation](https://mlsanity.com/paper/2503.06506/) (2025)
- [SONIC: Spectral Optimization of Noise for Inpainting with Consistency](https://mlsanity.com/paper/2511.19985/) (2025)
- [FastInit: Fast Noise Initialization for Temporally Consistent Video Generation](https://mlsanity.com/paper/2506.16119/) (2025)
- [Noise-Level Diffusion Guidance: Well Begun is Half Done](https://mlsanity.com/paper/2509.13936/) (2025)
- [CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration](https://mlsanity.com/paper/2509.17458/) (2025)
- [Demystifying Numerosity in Diffusion Models -- Limitations and Remedies](https://mlsanity.com/paper/2510.11117/) (2025)
- [Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer](https://mlsanity.com/paper/2511.19778/) (2025)
- [Towards Source-Aware Object Swapping with Initial Noise Perturbation](https://mlsanity.com/paper/2602.23697/) (2026)
- [LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation](https://mlsanity.com/paper/2603.24086/) (2026)
- [A Universal Avoidance Method for Diverse Multi-branch Generation](https://mlsanity.com/paper/2604.17323/) (2026)
- [LENS: Low-Frequency Eigen Noise Shaping for Efficient Diffusion Sampling](https://mlsanity.com/paper/2605.07253/) (2026)
- [When Does High-CFG Diffusion Inversion Fail? A Controlled Study of Prompt--Latent Interactions](https://mlsanity.com/paper/2607.04731/) (2026)
- [NoisEasier: Test-Time Noise Optimization for Text-to-Video Generation](https://mlsanity.com/paper/2608.30194/) (2026)
- [ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation](https://mlsanity.com/paper/2505.20935/) (2025)
- [Rethinking Refinement: Correcting Generative Bias without Noise Injection](https://mlsanity.com/paper/2601.21182/) (2026)
- [General and Efficient Steering of Diffusion Models](https://mlsanity.com/paper/2602.11395/) (2026)
- [ATHENA: Adaptive Test-Time Steering for Improving Count Fidelity in Diffusion Models](https://mlsanity.com/paper/2603.19676/) (2026)
- [Boosting Text-to-Image Diffusion Models via Core Token Attention-Based Seed Selection](https://mlsanity.com/paper/2605.19532/) (2026)
- [The Hitchhiker's Guide to Monoculture: AI Homogenizes Syntax, Not (Necessarily) Semantics](https://mlsanity.com/paper/2607.13077/) (2026)
- [Panoptic Scene Program Diffusion Transformer](https://mlsanity.com/paper/2609.31780/) (2026)
- [Optimizing H-Graph Hybridization for Diffusion-Guided RRT](https://mlsanity.com/paper/2609.32897/) (2026)

## Abstract (arXiv)

Text-to-image diffusion models have demonstrated remarkable capability in generating realistic images from arbitrary text prompts. However, they often produce inconsistent results for compositional prompts such as "two dogs" or "a penguin on the right of a bowl". Understanding these inconsistencies is crucial for reliable image generation. In this paper, we highlight the significant role of initial noise in these inconsistencies, where certain noise patterns are more reliable for compositional prompts than others. Our analyses reveal that different initial random seeds tend to guide the model to place objects in distinct image areas, potentially adhering to specific patterns of camera angles and image composition associated with the seed. To improve the model's compositional ability, we propose a method for mining these reliable cases, resulting in a curated training set of generated images without requiring any manual annotation. By fine-tuning text-to-image models on these generated images, we significantly enhance their compositional capabilities. For numerical composition, we observe relative increases of 29.3% and 19.5% for Stable Diffusion and PixArt-α, respectively. Spatial composition sees even larger gains, with 60.7% for Stable Diffusion and 21.1% for PixArt-α.