mlsanityData to 7 Oct 2026

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models

arXiv 2305.13655 ↗First posted 23 May 2023cs.CVcited by 242

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent advancements in text-to-image diffusion models have yielded impressive results in generating realistic and diverse images. However, these models still struggle with complex prompts, such as those that involve numeracy and spatial reasoning. This work proposes to enhance prompt understanding capabilities in diffusion models. Our method leverages a pretrained large language model (LLM) for grounded generation in a novel two-stage process. In the first stage, the LLM generates a scene layout that comprises captioned bounding boxes from a given prompt describing the desired image. In the second stage, a novel controller guides an off-the-shelf diffusion model for layout-grounded image generation. Both stages utilize existing pretrained models without additional model parameter optimization. Our method significantly outperforms the base diffusion model and several strong baselines in accurately generating images according to prompts that require various capabilities, doubling the generation accuracy across four tasks on average. Furthermore, our method enables instruction-based multi-round scene specification and can handle prompts in languages not supported by the underlying diffusion model. We anticipate that our method will unleash users' creativity by accurately following more complex prompts. Our code, demo, and benchmark are available at: https://llm-grounded-diffusion.github.io

Key facts. Cited by 242 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 48 of its 84 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

48 of 84 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation30 Mar
  2. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation16 Feb
  3. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  6. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  8. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  9. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2020Denoising Diffusion Implicit Modelscited by 9,655
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
25 more
  1. 2023Segment Anythingcited by 8,158
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  9. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  12. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  13. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  14. 2022Understanding Diffusion Models: A Unified Perspectivecited by 337
  15. 2018Image Generation from Scene Graphscited by 324
  16. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  17. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  18. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 225
  19. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  20. 2021Simple multi-dataset detectioncited by 126
  21. 2020LayoutTransformer: Layout Generation and Completion with Self-attentioncited by 122
  22. 2018Image Generation from Layoutcited by 105
  23. 2019Learning Canonical Representations for Scene Graph to Image Generationcited by 60
  24. 2022BlobGAN: Spatially Disentangled Scene Representationscited by 35
  25. 2021Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?cited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 32; 2024: 96; 2025: 73; 2026: 41.

242 papers.

Sort:

By area:

  1. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  2. 2024The Platonic Representation Hypothesiscited by 362
  3. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  5. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  6. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  7. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  8. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  9. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  10. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  11. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  12. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
38 more
  1. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  2. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  3. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  4. 2023LLM-grounded Video Diffusion Modelscited by 83
  5. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  6. 2023Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Followingcited by 77
  7. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 77
  8. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  9. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  10. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  11. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  12. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  13. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  14. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  15. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  16. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 58
  17. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 53
  18. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  19. 2024MagicTime: Time-lapse Video Generation Models as Metamorphic Simulatorscited by 51
  20. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  21. 2024Understanding the Impact of Negative Prompts: When and How Do They Take Effect?cited by 50
  22. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  23. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
  24. 2024MonoFormer: One Transformer for Both Diffusion and Autoregressioncited by 45
  25. 2024PromptFix: You Prompt and We Fix the Photocited by 40
  26. 2023Paragraph-to-Image Generation with Information-Enriched Diffusion Modelcited by 39
  27. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 39
  28. 2023AutoStory: Generating Diverse Storytelling Images with Minimal Human Effortcited by 38
  29. 2024Compositional Text-to-Image Generation with Dense Blob Representationscited by 38
  30. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Modelscited by 37
  31. 2023Real-Fake: Effective Training Data Synthesis Through Distribution Matchingcited by 35
  32. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  33. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  34. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  35. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  36. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  37. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seedscited by 31
  38. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 31

50 most cited of 242 All 242 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  2. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  3. 2023Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis
  4. 2023DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation
  5. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion
  6. 2024LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation
  7. 2023In-Context Learning Unlocked for Diffusion Models
  8. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception