mlsanityData to 14 Oct 2026

Self-correcting LLM-controlled Diffusion Models

arXiv 2311.16090 ↗First posted 27 Nov 2023cs.CVcited by 113

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image generation has witnessed significant progress with the advent of diffusion models. Despite the ability to generate photorealistic images, current text-to-image diffusion models still often struggle to accurately interpret and follow complex input text prompts. In contrast to existing models that aim to generate images only with their best effort, we introduce Self-correcting LLM-controlled Diffusion (SLD). SLD is a framework that generates an image from the input prompt, assesses its alignment with the prompt, and performs self-corrections on the inaccuracies in the generated image. Steered by an LLM controller, SLD turns text-to-image generation into an iterative closed-loop process, ensuring correctness in the resulting image. SLD is not only training-free but can also be seamlessly integrated with diffusion models behind API access, such as DALL-E 3, to further boost the performance of state-of-the-art diffusion models. Experimental results show that our approach can rectify a majority of incorrect generations, particularly in generative numeracy, attribute binding, and spatial relationships. Furthermore, by simply adjusting the instructions to the LLM, SLD can perform image editing tasks, bridging the gap between text-to-image generation and image editing pipelines. We will make our code available for future research and applications.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 30 of its 40 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

30 of 40 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts16 Oct
  2. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  3. 2023Controllable Text-to-Image Generation with GPT-429 May
  4. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  5. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Models20 Mar
  6. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  8. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  9. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  11. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
1 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 17

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  8. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  9. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  10. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  11. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  12. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
5 more
  1. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  2. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  3. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 179
  4. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  5. 2023LLM-grounded Video Diffusion Modelscited by 83

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 34; 2025: 45; 2026: 34.

113 papers.

Sort:

By area:

  1. 2024The Platonic Representation Hypothesiscited by 382
  2. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  3. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
  4. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 80
  5. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 61
  6. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 60
  7. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priorscited by 54
  8. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  9. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 43
  10. 2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewritingcited by 43
  11. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 41
  12. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 37
38 more
  1. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  2. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 33
  3. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  4. 2025Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generationcited by 30
  5. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 28
  6. 2025WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agentscited by 21
  7. 2025Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Drivingcited by 19
  8. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  9. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesiscited by 16
  10. 2025Test-time Prompt Refinement for Text-to-Image Modelscited by 16
  11. 2024VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correctioncited by 15
  12. 2024Self-Correcting Text-to-Video Generation with Misalignment Detection and Localized Refinementcited by 14
  13. 2025Agentic 3D Scene Generation with Spatially Contextualized VLMscited by 14
  14. 2025Dual-Process Image Generationcited by 12
  15. 2024MuLan: Multimodal-LLM Agent for Progressive and Interactive Multi-Object Diffusioncited by 11
  16. 2024Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Modelscited by 9
  17. 2024Free2Guide: Training-Free Text-to-Video Alignment using Image LVLMcited by 9
  18. 2024MUSES: 3D-Controllable Image Generation via Multi-Modal Agent Collaborationcited by 8
  19. 2025CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-stepcited by 8
  20. 2024Type-R: Automatically Retouching Typos for Text-to-Image Generationcited by 7
  21. 2025MoEdit: On Learning Quantity Perception for Multi-object Image Editingcited by 7
  22. 2025Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioningcited by 7
  23. 2026Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Modelscited by 7
  24. 2026coDrawAgents: A Multi-Agent Dialogue Framework for Compositional Image Generationcited by 7
  25. 2024Addressing Image Hallucination in Text-to-Image Generation through Factual Image Retrievalcited by 6
  26. 2024Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouchingcited by 6
  27. 2024What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generationcited by 6
  28. 2025VisualPrompter: Semantic-Aware Prompt Optimization with Visual Feedback for Text-to-Image Synthesiscited by 6
  29. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  30. 2026SOAR: Self-Correction for Optimal Alignment and Refinement in Diffusion Modelscited by 6
  31. 2025Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Modelscited by 5
  32. 2025Guiding What Not to Generate: Automated Negative Prompting for Text-Image Alignmentcited by 5
  33. 2024Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Languagecited by 4
  34. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspectivecited by 4
  35. 2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesiscited by 4
  36. 2025Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decompositioncited by 4
  37. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4
  38. 2025Implicit Bias Injection Attacks against Text-to-Image Diffusion Modelscited by 4

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey
  2. 2025LLMControl: Grounded Control of Text-to-Image Diffusion-based Synthesis with Multimodal LLMs
  3. 2026Training-Free Self-Correction for Multimodal Masked Diffusion Models
  4. 2026Towards Controllable Image Generation through Representation-Conditioned Diffusion Models
  5. 2023Text-to-image Diffusion Models in Generative AI: A Survey
  6. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selection
  7. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  8. 2023Filter-Guided Diffusion for Controllable Image Generation