# I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing

arXiv: https://arxiv.org/abs/2601.03741 · HTML: https://mlsanity.com/paper/2601.03741/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Builds on (most cited first, 26 of 26)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea](https://mlsanity.com/paper/2411.15738/) (2024)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer](https://mlsanity.com/paper/2504.20690/) (2025)
- [Transparent Image Layer Diffusion using Latent Transparency](https://mlsanity.com/paper/2402.17113/) (2024)
- [DiT4Edit: Diffusion Transformer for Image Editing](https://mlsanity.com/paper/2411.03286/) (2024)
- [pix2gestalt: Amodal Segmentation by Synthesizing Wholes](https://mlsanity.com/paper/2401.14398/) (2024)
- [GPT-IMAGE-EDIT-1.5M: A Million-Scale, GPT-Generated Image Dataset](https://mlsanity.com/paper/2507.21033/) (2025)
- [OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Data](https://mlsanity.com/paper/2501.07397/) (2025)
- [Instance-wise Occlusion and Depth Orders in Natural Scenes](https://mlsanity.com/paper/2111.14562/) (2021)
- [Alignment for Efficient Tool Calling of Large Language Models](https://mlsanity.com/paper/2503.06708/) (2025)
- [Image Editing As Programs with Diffusion Models](https://mlsanity.com/paper/2506.04158/) (2025)
- [Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering](https://mlsanity.com/paper/2511.23231/) (2025)
- [Addressing Text Embedding Leakage in Diffusion-based Image Editing](https://mlsanity.com/paper/2412.04715/) (2024)
- [MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents](https://mlsanity.com/paper/2512.00756/) (2025)

## Built on by (most cited first, 5 of 5)

- [Why Do DiT Editors Drift? Plug-and-Play Low Frequency Alignment in VAE Latent Space](https://mlsanity.com/paper/2605.08250/) (2026)
- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)
- [DataEvolver: Let Your Data Build and Improve Itself via Goal-Driven Loop Agents](https://mlsanity.com/paper/2605.01789/) (2026)
- [TopoAgent: An Agentic Framework for Automated Topology Learning in Medical Imaging](https://mlsanity.com/paper/2606.29763/) (2026)
- [Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets](https://mlsanity.com/paper/2610.04381/) (2026)

## Abstract (arXiv)

Existing text-guided image editing methods primarily rely on end-to-end pixel-level inpainting paradigm. Despite its success in simple scenarios, this paradigm still significantly struggles with compositional editing tasks that require precise local control and complex multi-object spatial reasoning. This paradigm is severely limited by 1) the implicit coupling of planning and execution, 2) the lack of object-level control granularity, and 3) the reliance on unstructured, pixel-centric modeling. To address these limitations, we propose I2E, a novel "Decompose-then-Action" paradigm that revisits image editing as an actionable interaction process within a structured environment. I2E utilizes a Decomposer to transform unstructured images into discrete, manipulable object layers and then introduces a physics-aware Vision-Language-Action Agent to parse complex instructions into a series of atomic actions via Chain-of-Thought reasoning. Further, we also construct I2E-Bench, a benchmark designed for multi-instance spatial reasoning and high-precision editing. Experimental results on I2E-Bench and multiple public benchmarks demonstrate that I2E significantly outperforms state-of-the-art methods in handling complex compositional instructions, maintaining physical plausibility, and ensuring multi-turn editing stability.