# An LLM-LVLM Driven Agent for Iterative and Fine-Grained Image Editing arXiv: https://arxiv.org/abs/2508.17435 ยท HTML: https://mlsanity.com/paper/2508.17435/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 22 of its 27 references that are in the map. - First posted on arXiv 24 Aug 2025. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 22 of 22) - [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022) - [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023) - [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024) - [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019) - [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025) - [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024) - [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024) - [Image inpainting: A review](https://mlsanity.com/paper/1909.06399/) (2019) - [Text to Image Generation with Semantic-Spatial Aware GAN](https://mlsanity.com/paper/2104.00567/) (2021) - [A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2406.14555/) (2024) - [Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback](https://mlsanity.com/paper/2501.01377/) (2025) - [EventBERT: A Pre-Trained Model for Event Correlation Reasoning](https://mlsanity.com/paper/2110.06533/) (2021) - [ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systems](https://mlsanity.com/paper/2312.06573/) (2023) - [Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation](https://mlsanity.com/paper/2505.24787/) (2025) - [Fine-Grained Distillation for Long Document Retrieval](https://mlsanity.com/paper/2212.10423/) (2022) - [Style-Aware Contrastive Learning for Multi-Style Image Captioning](https://mlsanity.com/paper/2301.11367/) (2023) - [A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops](https://mlsanity.com/paper/2412.17149/) (2024) - [ConceptAgent: LLM-Driven Precondition Grounding and Tree Search for Robust Task Planning and Execution](https://mlsanity.com/paper/2410.06108/) (2024) - [Conversation Routines: A Prompt Engineering Framework for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2501.11613/) (2025) - [Text to Image Generation and Editing: A Survey](https://mlsanity.com/paper/2505.02527/) (2025) - [Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editing](https://mlsanity.com/paper/2410.11374/) (2024) - [Agentic LLM Framework for Adaptive Decision Discourse](https://mlsanity.com/paper/2502.10978/) (2025) ## Built on by (most cited first, 8 of 8) - [MIRA: Multimodal Iterative Reasoning Agent for Image Editing](https://mlsanity.com/paper/2511.21087/) (2025) - [GenClaw: Code-Driven Agentic Image Generation](https://mlsanity.com/paper/2605.30248/) (2026) - [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026) - [Med-Banana: Learning Quality-Controlled Medical Image Editing from Success-and-Failure Trajectories](https://mlsanity.com/paper/2511.00801/) (2025) - [Making Implicit Preservation Intent Explicit in Conversational Image Editing](https://mlsanity.com/paper/2607.07051/) (2026) - [EditHero: A Benchmark for Long-Horizon Part-Level 3D Editing and Vibe Modeling](https://mlsanity.com/paper/2610.02298/) (2026) - [Reasoning-Informed Visual Editing](https://mlsanity.com/paper/2610.12343/) (2026) - [BluePixel: Toward Native 4K Agentic Image Editing](https://mlsanity.com/paper/2610.19882/) (2026) ## Abstract (arXiv) Despite the remarkable capabilities of text-to-image (T2I) generation models, real-world applications often demand fine-grained, iterative image editing that existing methods struggle to provide. Key challenges include granular instruction understanding, robust context preservation during modifications, and the lack of intelligent feedback mechanisms for iterative refinement. This paper introduces RefineEdit-Agent, a novel, training-free intelligent agent framework designed to address these limitations by enabling complex, iterative, and context-aware image editing. RefineEdit-Agent leverages the powerful planning capabilities of Large Language Models (LLMs) and the advanced visual understanding and evaluation prowess of Vision-Language Large Models (LVLMs) within a closed-loop system. Our framework comprises an LVLM-driven instruction parser and scene understanding module, a multi-level LLM-driven editing planner for goal decomposition, tool selection, and sequence generation, an iterative image editing module, and a crucial LVLM-driven feedback and evaluation loop. To rigorously evaluate RefineEdit-Agent, we propose LongBench-T2I-Edit, a new benchmark featuring 500 initial images with complex, multi-turn editing instructions across nine visual dimensions. Extensive experiments demonstrate that RefineEdit-Agent significantly outperforms state-of-the-art baselines, achieving an average score of 3.67 on LongBench-T2I-Edit, compared to 2.29 for Direct Re-Prompting, 2.91 for InstructPix2Pix, 3.16 for GLIGEN-based Edit, and 3.39 for ControlNet-XL. Ablation studies, human evaluations, and analyses of iterative refinement, backbone choices, tool usage, and robustness to instruction complexity further validate the efficacy of our agentic design in delivering superior edit fidelity and context preservation.