I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing
Not on the map. Why?
Abstract (arXiv)
Existing text-guided image editing methods primarily rely on end-to-end pixel-level inpainting paradigm. Despite its success in simple scenarios, this paradigm still significantly struggles with compositional editing tasks that require precise local control and complex multi-object spatial reasoning. This paradigm is severely limited by 1) the implicit coupling of planning and execution, 2) the lack of object-level control granularity, and 3) the reliance on unstructured, pixel-centric modeling. To address these limitations, we propose I2E, a novel "Decompose-then-Action" paradigm that revisits image editing as an actionable interaction process within a structured environment. I2E utilizes a Decomposer to transform unstructured images into discrete, manipulable object layers and then introduces a physics-aware Vision-Language-Action Agent to parse complex instructions into a series of atomic actions via Chain-of-Thought reasoning. Further, we also construct I2E-Bench, a benchmark designed for multi-instance spatial reasoning and high-precision editing. Experimental results on I2E-Bench and multiple public benchmarks demonstrate that I2E significantly outperforms state-of-the-art methods in handling complex compositional instructions, maintaining physical plausibility, and ensuring multi-turn editing stability.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 26 of its 45 references that are in the map. First posted on arXiv 7 Jan 2026.
Builds on
26 of 45 references.
References · 26
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
- 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
- 2018World Modelscited by 1,444
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
- 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
- 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
14 more
- 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
- 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 218
- 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
- 2024Transparent Image Layer Diffusion using Latent Transparencycited by 111
- 2024DiT4Edit: Diffusion Transformer for Image Editingcited by 100
- 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
- 2025GPT-IMAGE-EDIT-1.5M: A Million-Scale, GPT-Generated Image Datasetcited by 68
- 2025OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Datacited by 35
- 2021Instance-wise Occlusion and Depth Orders in Natural Scenescited by 31
- 2025Alignment for Efficient Tool Calling of Large Language Modelscited by 17
- 2025Image Editing As Programs with Diffusion Modelscited by 10
- 2024Addressing Text Embedding Leakage in Diffusion-based Image Editingcited by 7
- 2025Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineeringcited by 7
- 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agentscited by 5
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Why Do DiT Editors Drift? Plug-and-Play Low Frequency Alignment in VAE Latent Spacecited by 3
- 2026DataEvolver: Let Your Data Build and Improve Itself via Goal-Driven Loop Agentscited by 0
- 2026TopoAgent: An Agentic Framework for Automated Topology Learning in Medical Imagingcited by 0
- 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 0
- 2026Agentic Visual Generation: From Generative Models to Agentic Control6 Sep 2026
- 2026TopoAgent: An Agentic Framework for Automated Topology Learning in Medical Imaging29 Jun 2026
- 2026Why Do DiT Editors Drift? Plug-and-Play Low Frequency Alignment in VAE Latent Space7 May 2026
- 2026DataEvolver: Let Your Data Build and Improve Itself via Goal-Driven Loop Agents3 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026IMAGAgent: Orchestrating Multi-Turn Image Editing via Constraint-Aware Planning and Reflection
- 2026Agent Banana: High-Fidelity Image Editing with Agentic Thinking and Tooling
- 2026Making Image Editing Easier via Adaptive Task Reformulation with Agentic Executions
- 2026PhysEdit: Physically-Consistent Region-Aware Image Editing via Adaptive Spatio-Temporal Reasoning
- 2025ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependencies
- 2025CompBench: Benchmarking Complex Instruction-guided Image Editing
- 2026MCIE: Multimodal LLM-Driven Complex Instruction Image Editing with Spatial Guidance
- 2026Rethinking Where to Edit: Task-Aware Localization for Instruction-Based Image Editing