# Rethinking Where to Edit: Task-Aware Localization for Instruction-Based Image Editing

arXiv: https://arxiv.org/abs/2604.20258 · HTML: https://mlsanity.com/paper/2604.20258/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Builds on (most cited first, 28 of 28)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions](https://mlsanity.com/paper/2104.14806/) (2021)
- [Splicing ViT Features for Semantic Appearance Transfer](https://mlsanity.com/paper/2201.00424/) (2022)
- [VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://mlsanity.com/paper/2312.14867/) (2023)
- [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://mlsanity.com/paper/2310.01506/) (2023)
- [ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling](https://mlsanity.com/paper/2501.02487/) (2025)
- [InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions](https://mlsanity.com/paper/2305.18047/) (2023)
- [ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer](https://mlsanity.com/paper/2410.00086/) (2024)
- [Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations](https://mlsanity.com/paper/2505.18584/) (2025)
- [Lumina-OmniLV: A Unified Multimodal Framework for General Low-Level Vision](https://mlsanity.com/paper/2504.04903/) (2025)
- [EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing](https://mlsanity.com/paper/2509.13399/) (2025)
- [Group Relative Attention Guidance for Image Editing](https://mlsanity.com/paper/2510.24657/) (2025)

## Abstract (arXiv)

Instruction-based image editing (IIE) aims to modify an image according to a natural language instruction. Despite recent advances in diffusion transformers, existing methods often introduce unintended changes to regions unrelated to the requested edit. We attribute this limitation to the absence of an explicit mechanism for edit localization. Different editing operations (e.g., subject addition, removal, and replacement) induce distinct spatial patterns, yet existing IIE models typically perform localization in a task-agnostic manner. To address this limitation, we propose a training-free, task-aware edit localization framework that exploits the intrinsic source and target image streams of IIE models. For each image stream, we construct feature centroids from attention-based edit cues, and then partition tokens into edit and non-edit regions based on feature similarity. Observing that effective localization is inherently task-dependent, we introduce a unified mask construction strategy that selectively leverages the source and target streams according to the editing task. We also provide a systematic analysis of our underlying insights and design choices. Extensive experiments on EdiVal-Bench demonstrate that our framework consistently improves content consistency in non-edit regions while maintaining strong instruction-following performance on top of Step1X-Edit and Qwen-Image-Edit.