# Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks

arXiv: https://arxiv.org/abs/2401.07709 · HTML: https://mlsanity.com/paper/2401.07709/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 15 Jan 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Localizing Object-level Shape Variations with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.11306/) (2023)
- [Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance](https://mlsanity.com/paper/2210.05559/) (2022)
- [FlexIT: Towards Flexible Semantic Image Translation](https://mlsanity.com/paper/2203.04705/) (2022)
- [ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generation](https://mlsanity.com/paper/2204.04428/) (2022)

## Built on by (most cited first, 16 of 16)

- [IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout](https://mlsanity.com/paper/2506.01949/) (2025)
- [Benchmarking Counterfactual Image Generation](https://mlsanity.com/paper/2403.20287/) (2024)
- [VINCIE: Unlocking In-context Image Editing from Video](https://mlsanity.com/paper/2506.10941/) (2025)
- [Addressing Text Embedding Leakage in Diffusion-based Image Editing](https://mlsanity.com/paper/2412.04715/) (2024)
- [ModelLock: Locking Your Model With a Spell](https://mlsanity.com/paper/2405.16285/) (2024)
- [AdaFlow: Efficient Long Video Editing via Adaptive Attention Slimming And Keyframe Selection](https://mlsanity.com/paper/2502.05433/) (2025)
- [Click2Mask: Local Editing with Dynamic Mask Generation](https://mlsanity.com/paper/2409.08272/) (2024)
- [MUSE: Manipulating Unified Framework for Synthesizing Emotions in Images via Test-Time Optimization](https://mlsanity.com/paper/2511.21051/) (2025)
- [Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-training](https://mlsanity.com/paper/2603.16139/) (2026)
- [Physics-informed DeepCT: Sinogram Wavelet Decomposition Meets Masked Diffusion](https://mlsanity.com/paper/2501.09935/) (2025)
- [From Competition to Coopetition: Coopetitive Training-Free Image Editing Based on Text Guidance](https://mlsanity.com/paper/2604.15948/) (2026)
- [MFGDiffusion: Mask-Guided Smoke Synthesis for Enhanced Forest Fire Detection](https://mlsanity.com/paper/2507.11252/) (2025)
- [DirectSwap: Paired, Mask-Free Video Head Swapping with Full-Reference Evaluation](https://mlsanity.com/paper/2512.09417/) (2025)
- [LAMS-Edit: Latent and Attention Mixing with Schedulers for Improved Content Preservation in Diffusion-Based Image and Style Editing](https://mlsanity.com/paper/2601.02987/) (2026)
- [Controllable Diffusion-Based Lesion Inpainting for Scalable Histopathology Data Augmentation](https://mlsanity.com/paper/2601.08127/) (2026)
- [H-Adapter: Pose-Robust Hairstyle Transfer via Attention-Derived, Source-Aligned Hair Masks](https://mlsanity.com/paper/2606.25578/) (2026)

## Abstract (arXiv)

Diffusion-based Image Editing (DIE) is an emerging research hot-spot, which often applies a semantic mask to control the target area for diffusion-based editing. However, most existing solutions obtain these masks via manual operations or off-line processing, greatly reducing their efficiency. In this paper, we propose a novel and efficient image editing method for Text-to-Image (T2I) diffusion models, termed Instant Diffusion Editing(InstDiffEdit). In particular, InstDiffEdit aims to employ the cross-modal attention ability of existing diffusion models to achieve instant mask guidance during the diffusion steps. To reduce the noise of attention maps and realize the full automatics, we equip InstDiffEdit with a training-free refinement scheme to adaptively aggregate the attention distributions for the automatic yet accurate mask generation. Meanwhile, to supplement the existing evaluations of DIE, we propose a new benchmark called Editing-Mask to examine the mask accuracy and local editing ability of existing methods. To validate InstDiffEdit, we also conduct extensive experiments on ImageNet and Imagen, and compare it with a bunch of the SOTA methods. The experimental results show that InstDiffEdit not only outperforms the SOTA methods in both image quality and editing results, but also has a much faster inference speed, i.e., +5 to +6 times.