mlsanityData to 7 Oct 2026

Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation

arXiv 2310.08541 ↗First posted 12 Oct 2023cs.CVcited by 38

Not on the map. Why?

Abstract (arXiv)

We introduce ``Idea to Image,'' a system that enables multimodal iterative self-refinement with GPT-4V(ision) for automatic image design and generation. Humans can quickly identify the characteristics of different text-to-image (T2I) models via iterative explorations. This enables them to efficiently convert their high-level generation ideas into effective T2I prompts that can produce good images. We investigate if systems based on large multimodal models (LMMs) can develop analogous multimodal self-refinement abilities that enable exploring unknown models or environments via self-refining tries. Idea2Img cyclically generates revised T2I prompts to synthesize draft images, and provides directional feedback for prompt revision, both conditioned on its memory of the probed T2I model's characteristics. The iterative self-refinement brings Idea2Img various advantages over vanilla T2I models. Notably, Idea2Img can process input ideas with interleaved image-text sequences, follow ideas with design instructions, and generate images of better semantic and visual qualities. The user preference study validates the efficacy of multimodal iterative self-refinement on automatic image design and generation.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 54 of its 67 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

54 of 67 references.

References · 54

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
38 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  10. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  12. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  13. 2023Teaching Large Language Models to Self-Debugcited by 963
  14. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  15. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  16. 2023Large Language Models as Optimizerscited by 830
  17. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  18. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  19. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  20. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  21. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  22. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  23. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  24. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  25. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  26. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  27. 2015A Neural Algorithm of Artistic Stylecited by 480
  28. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  29. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  30. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  31. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  32. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  33. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  34. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  35. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  36. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  37. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  38. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 15; 2025: 13; 2026: 4.

38 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  3. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  4. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  5. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  6. 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 81
  7. 2025LEGION: Learning to Ground and Explain for Synthetic Image Detectioncited by 65
  8. 2023COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Designcited by 39
  9. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  10. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  11. 2024SketchAgent: Language-Driven Sequential Sketch Generationcited by 30
  12. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
26 more
  1. 2023How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigationcited by 22
  2. 2023DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Designcited by 17
  3. 2025Talk Structurally, Act Hierarchically: A Collaborative Framework for LLM Multi-Agent Systemscited by 17
  4. 2025Multi-turn Consistent Image Editingcited by 17
  5. 2025ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generationcited by 14
  6. 2024VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoningcited by 13
  7. 2024Automated Black-box Prompt Engineering for Personalized Text-to-Image Generationcited by 12
  8. 2024Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputscited by 12
  9. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  10. 2024SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editingcited by 10
  11. 2024CCA: Collaborative Competitive Agents for Image Editingcited by 9
  12. 2024Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Modelscited by 4
  13. 2025Marmot: Object-Level Self-Correction via Multi-Agent Reasoningcited by 3
  14. 2025POET: Preference Optimization for Enhanced Text-to-Image Generationcited by 3
  15. 2026OctoT2I: A Self-Evolving Agentic Text-to-Image Routercited by 3
  16. 2024GenSERP: Large Language Models for Whole Page Presentationcited by 1
  17. 2024Bridging the Intent Gap: Knowledge-Enhanced Visual Generationcited by 1
  18. 2025OSPO: Object-Centric Self-Improving Preference Optimization for Text-to-Image Generationcited by 1
  19. 2025Mirror in the Model: Ad Banner Image Generation via Reflective Multi-LLM and Multi-modal Agentscited by 1
  20. 20253DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMscited by 1
  21. 20263D Space as a Scratchpad for Editable Text-to-Image Generationcited by 1
  22. 2026Agentic Flow Steering and Parallel Rollout Search for Spatially Grounded Text-to-Image Generationcited by 1
  23. 2023Visual AI and Linguistic Intelligence Through Steerability and Composabilitycited by 0
  24. 2025ESCT3D: Efficient and Selectively Controllable Text-Driven 3D Content Generation with Gaussian Splattingcited by 0
  25. 2025Counteracting Matthew Effect in Self-Improvement of LVLMs through Head-Tail Re-balancingcited by 0
  26. 2026Editable Visual Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis
  2. 2026Iterative Refinement Improves Compositional Image Generation
  3. 2025Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation
  4. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestration
  5. 2025CRAFT: Continuous Reasoning and Agentic Feedback Tuning for Multimodal Text-to-Image Generation
  6. 2025Test-time Prompt Refinement for Text-to-Image Models
  7. 2024GPTDrawer: Enhancing Visual Synthesis through ChatGPT
  8. 2025Improving Chain-of-Thought Efficiency for Autoregressive Image Generation