mlsanityData to 7 Oct 2026

Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs

arXiv 2401.11708 ↗First posted 22 Jan 2024cs.CV cs.AI cs.LGcited by 224

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

also in 1 more
Abstract (arXiv)

Diffusion models have exhibit exceptional performance in text-to-image generation and editing. However, existing methods often face challenges when handling complex text prompts that involve multiple objects with multiple attributes and relationships. In this paper, we propose a brand new training-free text-to-image generation/editing framework, namely Recaption, Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning ability of multimodal LLMs to enhance the compositionality of text-to-image diffusion models. Our approach employs the MLLM as a global planner to decompose the process of generating complex images into multiple simpler generation tasks within subregions. We propose complementary regional diffusion to enable region-wise compositional generation. Furthermore, we integrate text-guided image generation and editing within the proposed RPG in a closed-loop fashion, thereby enhancing generalization ability. Extensive experiments demonstrate our RPG outperforms state-of-the-art text-to-image diffusion models, including DALL-E 3 and SDXL, particularly in multi-category object composition and text-image semantic alignment. Notably, our RPG framework exhibits wide compatibility with various MLLM architectures (e.g., MiniGPT-4) and diffusion backbones (e.g., ControlNet). Our code is available at: https://github.com/YangLing0818/RPG-DiffusionMaster

Key facts. Cited by 224 papers in the map (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 70 of its 80 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

70 of 80 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Models29 Sep

Other references · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Segment Anythingcited by 8,158
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  13. 2020Learning to summarize from human feedbackcited by 2,562
  14. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  15. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  16. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  17. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  18. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  19. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  20. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  21. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  22. 2022Galactica: A Large Language Model for Sciencecited by 736
  23. 2016Generative Adversarial Text to Image Synthesiscited by 735
  24. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  25. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  26. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  27. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  28. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  29. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  30. 2017Generative Adversarial Networks: An Overviewcited by 588
  31. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  32. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  33. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  34. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  35. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  36. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  37. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  38. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 83; 2025: 84; 2026: 57.

224 papers.

Sort:

By area:

  1. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  2. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  3. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  4. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  6. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  7. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  8. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  9. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  10. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  11. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  12. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
38 more
  1. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  2. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  3. 2024Consistency Flow Matching: Defining Straight Flows with Velocity Consistencycited by 101
  4. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 78
  5. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 77
  6. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  7. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  8. 2025SeedEdit 3.0: Fast and High-Quality Generative Image Editingcited by 52
  9. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  10. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Modelscited by 47
  11. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  12. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 46
  13. 2024Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generationcited by 43
  14. 2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewritingcited by 38
  15. 2024EditWorld: Simulating World Dynamics for Instruction-Following Image Editingcited by 37
  16. 2024Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinementcited by 37
  17. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  18. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Modelscited by 37
  19. 2025PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Frameworkcited by 37
  20. 2024PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Modelscited by 36
  21. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  22. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  23. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  24. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  25. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  26. 2024LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generationcited by 30
  27. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  28. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  29. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  30. 2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 30
  31. 2024DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generationcited by 26
  32. 2024Training-free Regional Prompting for Diffusion Transformerscited by 26
  33. 2025Training-free Diffusion Acceleration with Bottleneck Samplingcited by 25
  34. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  35. 2024Large Language Models for Manufacturingcited by 24
  36. 2025MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generationcited by 23
  37. 2025Embodied AI: From LLMs to World Modelscited by 23
  38. 2025EditThinker: Unlocking Iterative Reasoning for Any Image Editorcited by 23

50 most cited of 224 All 224 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024TIE: Revolutionizing Text-based Image Editing for Complex-Prompt Following and High-Fidelity Editing
  2. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts
  3. 2026Reinforcing the Generation Order of Multimodal Masked Diffusion Models
  4. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion
  5. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  6. 2026Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoders
  7. 2023Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis
  8. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models