mlsanityData to 7 Oct 2026

GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing

arXiv 2407.05600 ↗First posted 8 Jul 2024cs.CVcited by 132

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Despite the success achieved by existing image generation and editing methods, current models still struggle with complex problems including intricate text prompts, and the absence of verification and self-correction mechanisms makes the generated images unreliable. Meanwhile, a single model tends to specialize in particular tasks and possess the corresponding capabilities, making it inadequate for fulfilling all user requirements. We propose GenArtist, a unified image generation and editing system, coordinated by a multimodal large language model (MLLM) agent. We integrate a comprehensive range of existing models into the tool library and utilize the agent for tool selection and execution. For a complex problem, the MLLM agent decomposes it into simpler sub-problems and constructs a tree structure to systematically plan the procedure of generation, editing, and self-correction with step-by-step verification. By automatically generating missing position-related inputs and incorporating position information, the appropriate tool can be effectively employed to address each sub-problem. Experiments demonstrate that GenArtist can perform various generation and editing tasks, achieving state-of-the-art performance and surpassing existing models such as SDXL and DALL-E 3, as can be seen in Fig. 1. Project page is https://zhenyuw16.github.io/GenArtist_page.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 62 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

58 of 62 references.

References · 58

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  6. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  7. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  8. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  9. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  10. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  11. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  12. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  13. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  14. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  15. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  16. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  17. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  18. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  19. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  20. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  21. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  22. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
  23. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  24. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  25. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
  26. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 290
  27. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  28. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  29. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  30. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  31. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  32. 2023TextDiffuser: Diffusion Models as Text Painterscited by 223
  33. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 193
  34. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  35. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  36. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  37. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  38. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 61; 2026: 58.

132 papers.

Sort:

By area:

  1. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  2. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  3. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  4. 2024Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usagecited by 49
  5. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
  6. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 39
  7. 2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewritingcited by 38
  8. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  9. 2024BrushEdit: All-In-One Image Inpainting and Editingcited by 31
  10. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 31
  11. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  12. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
38 more
  1. 2025Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generationcited by 30
  2. 2025ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generationcited by 20
  3. 2025MAViS: A Multi-Agent Framework for Long-Sequence Video Storytellingcited by 19
  4. 2026Agent Banana: High-Fidelity Image Editing with Agentic Thinking and Toolingcited by 17
  5. 2024DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generationcited by 16
  6. 2025ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedbackcited by 15
  7. 2025MIRA: Multimodal Iterative Reasoning Agent for Image Editingcited by 15
  8. 2025Agentic 3D Scene Generation with Spatially Contextualized VLMscited by 14
  9. 2025ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memorycited by 14
  10. 2025Talk2Image: A Multi-Agent System for Multi-Turn Image Generation and Editingcited by 13
  11. 2024ChatGen: Automatic Text-to-Image Generation From FreeStyle Chattingcited by 11
  12. 2024SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editingcited by 11
  13. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  14. 2025MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skillscited by 10
  15. 2025Image Editing As Programs with Diffusion Modelscited by 10
  16. 2024Edicho: Consistent Image Editing in the Wildcited by 9
  17. 2025Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editingcited by 9
  18. 2025MultiRef: Controllable Image Generation with Multiple Visual Referencescited by 9
  19. 2025UniShield: An Adaptive Multi-Agent Framework for Unified Forgery Image Detection and Localizationcited by 9
  20. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  21. 2025Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoningcited by 8
  22. 2025CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-stepcited by 8
  23. 2025Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editingcited by 8
  24. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 8
  25. 2025ToolForge: A Data Synthesis Pipeline for Multi-Hop Search without Real-World APIscited by 8
  26. 2026MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolutioncited by 8
  27. 2026VQQA: An Agentic Approach for Video Evaluation and Quality Improvementcited by 8
  28. 2025When Cultures Meet: Multicultural Text-to-Image Generationcited by 7
  29. 2025MoEdit: On Learning Quantity Perception for Multi-object Image Editingcited by 7
  30. 2025LayerEdit: Disentangled Multi-Object Editing via Conflict-Aware Multi-Layer Learningcited by 7
  31. 2025GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generationcited by 6
  32. 2026Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Modelscited by 6
  33. 2026coDrawAgents: A Multi-Agent Dialogue Framework for Compositional Image Generationcited by 6
  34. 2025CoSTAast: Cost-Sensitive Toolpath Agent for Multi-turn Image Editingcited by 5
  35. 2025See, Think, Act: Online Shopper Behavior Simulation with VLM Agentscited by 5
  36. 2025The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignmentcited by 5
  37. 2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesiscited by 4
  38. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4

50 most cited of 132 All 132 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LLMGA: Multimodal Large Language Model based Generation Assistant
  2. 2024LLMs Meet Multimodal Generation and Editing: A Survey
  3. 2025ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation
  4. 2026APE: Agentic Prompt Enhancer for Image Generation and Editing
  5. 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget
  6. 2025CRAFT: Continuous Reasoning and Agentic Feedback Tuning for Multimodal Text-to-Image Generation
  7. 2026DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing
  8. 2026GenAgent: Scaling Text-to-Image Generation via Agentic Multimodal Reasoning