From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing
Not on the map. Why?
Abstract (arXiv)
Modern image editing models produce realistic results but struggle with abstract, multi step instructions (e.g., ``make this advertisement more vegetarian-friendly''). Prior agent based methods decompose such tasks but rely on handcrafted pipelines or teacher imitation, limiting flexibility and decoupling learning from actual editing outcomes. We propose an experiential framework for long-horizon image editing, where a planner generates structured atomic decompositions and an orchestrator selects tools and regions to execute each step. A vision language judge provides outcome-based rewards for instruction adherence and visual quality. The orchestrator is trained to maximize these rewards, and successful trajectories are used to refine the planner. By tightly coupling planning with reward driven execution, our approach yields more coherent and reliable edits than single-step or rule-based multistep baselines.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 14 May 2026.
Builds on
51 of 59 references.
References · 51
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2023Visual Instruction Tuningcited by 7,869
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
- 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
38 more
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
- 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
- 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
- 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
- 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
- 2023Zero-shot Image-to-Image Translationcited by 505
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
- 2024OmniGen: Unified Image Generationcited by 394
- 2026Reinforcement Learning via Self-Distillationcited by 332
- 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
- 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 294
- 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
- 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
- 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
- 2023Style Aligned Image Generation via Shared Attentioncited by 216
- 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 201
- 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 195
- 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
- 2025DeepSeek-OCR: Contexts Optical Compressioncited by 183
- 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
- 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 173
- 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
- 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
- 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 66
- 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 64
- 2025EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editingcited by 62
- 2024Compositional Text-to-Image Generation with Dense Blob Representationscited by 38
- 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
- 2025Qwen-Image-Layered: Towards Inherent Editability via Layer Decompositioncited by 34
- 2025LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integrationcited by 19
- 2025MIRA: Multimodal Iterative Reasoning Agent for Image Editingcited by 17
- 2025Talk2Image: A Multi-Agent System for Multi-Turn Image Generation and Editingcited by 13
- 2025Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editingcited by 9
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.