mlsanityData to 7 Oct 2026

LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation

arXiv 2308.05095 ↗First posted 9 Aug 2023cs.CV cs.AIcited by 92

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

In the text-to-image generation field, recent remarkable progress in Stable Diffusion makes it possible to generate rich kinds of novel photorealistic images. However, current models still face misalignment issues (e.g., problematic spatial relation understanding and numeration failure) in complex natural scenes, which impedes the high-faithfulness text-to-image generation. Although recent efforts have been made to improve controllability by giving fine-grained guidance (e.g., sketch and scribbles), this issue has not been fundamentally tackled since users have to provide such guidance information manually. In this work, we strive to synthesize high-fidelity images that are semantically aligned with a given textual prompt without any guidance. Toward this end, we propose a coarse-to-fine paradigm to achieve layout planning and image generation. Concretely, we first generate the coarse-grained layout conditioned on a given textual prompt via in-context learning based on Large Language Models. Afterward, we propose a fine-grained object-interaction diffusion method to synthesize high-faithfulness images conditioned on the prompt and the automatically generated layout. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art models in terms of layout and image generation. Our code and settings are available at https://layoutllm-t2i.github.io.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 69 of its 89 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

69 of 89 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation30 Mar
  2. 2023LayoutDM: Discrete Diffusion Model for Controllable Layout Generation14 Mar
  3. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation16 Feb
  4. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  5. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  7. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  8. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  9. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  10. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
2 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  5. 2018On the Spectral Bias of Neural Networkscited by 1,321
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  9. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  10. 2016Generative Adversarial Text to Image Synthesiscited by 735
  11. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  12. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  13. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  14. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  15. 2018Image Generation from Scene Graphscited by 324
  16. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  17. 2022Active Example Selection for In-Context Learningcited by 195
  18. 2020LayoutTransformer: Layout Generation and Completion with Self-attentioncited by 122
  19. 2016Learning What and Where to Drawcited by 105
  20. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  21. 2021Unified Named Entity Recognition as Word-Word Relation Classificationcited by 89
  22. 2019Image Synthesis From Reconfigurable Layout and Stylecited by 85
  23. 2021Constrained Graphic Layout Generation via Latent Optimizationcited by 83
  24. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 79
  25. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  26. 2021BLT: Bidirectional Layout Transformer for Controllable Layout Generationcited by 69
  27. 2019READ: Recursive Autoencoders for Document Layout Generationcited by 51
  28. 2020Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpuscited by 49
  29. 2020Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Networkcited by 48
  30. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  31. 2019Generating Multiple Objects at Spatially Distinct Locationscited by 41
  32. 2021Context-Aware Layout to Image Generation with Enhanced Object Appearancecited by 41
  33. 2023X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidancecited by 37
  34. 2021Learn from Syntax: Improving Pair-wise Aspect and Opinion Terms Extractionwith Rich Syntactic Knowledgecited by 31
  35. 2023Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucinationcited by 27
  36. 2021Mastering the Explicit Opinion-role Interaction: Syntax-aided Neural Transition System for Unified Opinion Role Labelingcited by 25
  37. 2022Effective Token Graph Modeling using a Novel Labeling Strategy for Structured Sentiment Analysiscited by 22
  38. 2023Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modelingcited by 16

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 46; 2025: 27; 2026: 11.

92 papers.

Sort:

By area:

  1. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  2. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  4. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  5. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  6. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  7. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  8. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  9. 2023Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMscited by 35
  10. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
  11. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  12. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seedscited by 31
38 more
  1. 2024A Review of Multi-Modal Large Language and Vision Modelscited by 30
  2. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  3. 2024A New Creative Generation Pipeline for Click-Through Rate with Stable Diffusion Modelcited by 26
  4. 2024CityCraft: A Real Crafter for 3D City Generationcited by 25
  5. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  6. 2025MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generationcited by 23
  7. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Predictioncited by 20
  8. 2024AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generationcited by 20
  9. 2023Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generationcited by 19
  10. 2024PMG : Personalized Multimodal Generation with Large Language Modelscited by 19
  11. 2023SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Controlcited by 18
  12. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  13. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 18
  14. 2025VINCIE: Unlocking In-context Image Editing from Videocited by 17
  15. 2025Test-time Prompt Refinement for Text-to-Image Modelscited by 16
  16. 2024SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generationcited by 15
  17. 2024GroundingBooth: Grounding Text-to-Image Customizationcited by 13
  18. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Modelscited by 12
  19. 2024PerLDiff: Controllable Street View Synthesis Using Perspective-Layout Diffusion Modelscited by 12
  20. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11
  21. 2024Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editingcited by 10
  22. 2024SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editingcited by 10
  23. 2025MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generationcited by 8
  24. 2023ChatTraffic: Text-to-Traffic Generation via Diffusion Modelcited by 7
  25. 2024DreamRunner: Fine-Grained Compositional Story-to-Video Generation with Retrieval-Augmented Motion Adaptationcited by 7
  26. 2024CC-Diff: Enhancing Contextual Coherence in Remote Sensing Image Synthesiscited by 7
  27. 2024Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouchingcited by 6
  28. 2024Spider: Any-to-Many Multimodal LLMcited by 6
  29. 2026coDrawAgents: A Multi-Agent Dialogue Framework for Compositional Image Generationcited by 6
  30. 2024LoopAnimate: Loopable Salient Object Animationcited by 5
  31. 2024Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Languagecited by 4
  32. 2024GVDIFF: Grounded Text-to-Video Generation with Diffusion Modelscited by 4
  33. 2024Mixed-View Panorama Synthesis using Geospatially Guided Diffusioncited by 4
  34. 2024An Evaluation of Standard Statistical Models and LLMs on Time Series Forecastingcited by 4
  35. 2025Multi-Object Sketch Animation by Scene Decomposition and Motion Planningcited by 4
  36. 2025Detail++: Training-Free Detail Enhancer for T2I Diffusion Modelscited by 4
  37. 2025Steer Away From Mode Collisions: Improving Composition In Diffusion Modelscited by 4
  38. 2025TTOM: Test-Time Optimization and Memorization for Compositional Video Generationcited by 4

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis
  2. 2024Training-free Composite Scene Generation for Layout-to-Image Synthesis
  3. 2025A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models
  4. 2023SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation
  5. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis
  6. 2024Training-Free Layout-to-Image Generation with Marginal Attention Constraints
  7. 2026Benchmarking Layout-Guided Diffusion Models through Unified Semantic-Spatial Evaluation in Closed and Open Settings
  8. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts