IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Current image editing software often hinges on fixed filters or expert tuning, leaving a gap between amateur users' intent and outcomes. Creations by generative models may contain artifacts, implausible details, or stylistic drift away from photorealism and offer little insight into why an edit was made. We propose IEA, a conversational Image Editing Agent that learns to operate parameterized tools in an explicit, interpretable action space. IEA is trained via a three-stage multitask pipeline: (1) SFT on distilled expert edits, (2) GRPO with rewards for likeness improvement, tool usefulness, and intent summarization, and (3) large-scale synthetic fine-tuning to jointly master image editing, refinement, and user intent summarization. By manipulating 16 editing tools step by step, IEA produces transparent edit traces that can be inspected and debugged. In quantitative experiments, it attains a lower pixel distance on the edit task and a higher ROUGE-L on the summary task than strong baselines. In user studies, it ranks best among tool-calling methods for instruction following while surpassing generative methods in overall perceptual quality. Our results validate interpretable, tool-centric VLMs as a reliable path to human instruction-guided image retouching.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 69 of its 85 references that are in the map. First posted on arXiv 6 Jun 2026.
Builds on
69 of 85 references.
Closest ancestors · 3
Same direction, newest first
- 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent21 Jun
- 2025PhotoArtAgent: Intelligent Photo Retouching with Language Model-Based Artist Agents29 May
- 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing8 Jul
Other references · 66
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
38 more
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
- 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
- 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
- 2025Qwen-Image Technical Reportcited by 1,044
- 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
- 2023Large Language Modelscited by 780
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
- 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
- 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
- 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
- 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
- 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
- 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
- 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
- 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
- 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
- 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 177
- 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
- 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 134
- 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
- 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 105
- 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
- 2024TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Modelscited by 64
- 2024ToolNet: Connecting Large Language Models with Massive Tools via Tool Graphcited by 60
- 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
- 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
- 2018Distort-and-Recover: Color Enhancement using Deep Reinforcement Learningcited by 52
- 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
- 2024ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrievalcited by 41
50 most cited of 66 All 66 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026TalkPhoto: A Versatile Training-Free Conversational Assistant for Intelligent Image Editing
- 2023DialogPaint: A Dialog-based Image Editing Model
- 2026EditRefiner: A Human-Aligned Agentic Framework for Image Editing Refinement
- 2026What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systems
- 2025Image Editing As Programs with Diffusion Models
- 2025EditThinker: Unlocking Iterative Reasoning for Any Image Editor
- 2026ImageEdit-R1: Boosting Multi-Agent Image Editing via Reinforcement Learning
- 2026Making Image Editing Easier via Adaptive Task Reformulation with Agentic Executions