mlsanityData to 14 Oct 2026

IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment

arXiv 2606.08016 ↗First posted 6 Jun 2026cs.CV cs.AI cs.CLcited by 2

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Current image editing software often hinges on fixed filters or expert tuning, leaving a gap between amateur users' intent and outcomes. Creations by generative models may contain artifacts, implausible details, or stylistic drift away from photorealism and offer little insight into why an edit was made. We propose IEA, a conversational Image Editing Agent that learns to operate parameterized tools in an explicit, interpretable action space. IEA is trained via a three-stage multitask pipeline: (1) SFT on distilled expert edits, (2) GRPO with rewards for likeness improvement, tool usefulness, and intent summarization, and (3) large-scale synthetic fine-tuning to jointly master image editing, refinement, and user intent summarization. By manipulating 16 editing tools step by step, IEA produces transparent edit traces that can be inspected and debugged. In quantitative experiments, it attains a lower pixel distance on the edit task and a higher ROUGE-L on the summary task than strong baselines. In user studies, it ranks best among tool-calling methods for instruction following while surpassing generative methods in overall perceptual quality. Our results validate interpretable, tool-centric VLMs as a reliable path to human instruction-guided image retouching.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 69 of its 85 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

69 of 85 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent21 Jun
  2. 2025PhotoArtAgent: Intelligent Photo Retouching with Language Model-Based Artist Agents29 May
  3. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing8 Jul

Other references · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
38 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  6. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  12. 2025Qwen-Image Technical Reportcited by 1,044
  13. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  14. 2023Large Language Modelscited by 780
  15. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  16. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  17. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  18. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  19. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  20. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  21. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  22. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  23. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  24. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  25. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  26. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 177
  27. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  28. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 134
  29. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  30. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 105
  31. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  32. 2024TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Modelscited by 64
  33. 2024ToolNet: Connecting Large Language Models with Massive Tools via Tool Graphcited by 60
  34. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
  35. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  36. 2018Distort-and-Recover: Color Enhancement using Deep Reinforcement Learningcited by 52
  37. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  38. 2024ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrievalcited by 41

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026TalkPhoto: A Versatile Training-Free Conversational Assistant for Intelligent Image Editing
  2. 2023DialogPaint: A Dialog-based Image Editing Model
  3. 2026EditRefiner: A Human-Aligned Agentic Framework for Image Editing Refinement
  4. 2026What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systems
  5. 2025Image Editing As Programs with Diffusion Models
  6. 2025EditThinker: Unlocking Iterative Reasoning for Any Image Editor
  7. 2026ImageEdit-R1: Boosting Multi-Agent Image Editing via Reinforcement Learning
  8. 2026Making Image Editing Easier via Adaptive Task Reformulation with Agentic Executions