# LIME: Localized Image Editing via Attention Regularization in Diffusion Models

arXiv: https://arxiv.org/abs/2312.09256 · HTML: https://mlsanity.com/paper/2312.09256/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 55 of its 62 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)

## Built on by (most cited first, 27 of 27)

- [Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance](https://mlsanity.com/paper/2403.17377/) (2024)
- [Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions](https://mlsanity.com/paper/2403.17064/) (2024)
- [Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models](https://mlsanity.com/paper/2508.12880/) (2025)
- [Qwen-Image-Layered: Towards Inherent Editability via Layer Decomposition](https://mlsanity.com/paper/2512.15603/) (2025)
- [SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing](https://mlsanity.com/paper/2505.02370/) (2025)
- [SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing](https://mlsanity.com/paper/2404.05717/) (2024)
- [InstructBrush: Learning Attention-based Instruction Optimization for Image Editing](https://mlsanity.com/paper/2403.18660/) (2024)
- [ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval](https://mlsanity.com/paper/2505.20764/) (2025)
- [WorldEdit: Towards Open-World Image Editing with a Knowledge-Informed Benchmark](https://mlsanity.com/paper/2602.07095/) (2026)
- [FunEditor: Achieving Complex Image Edits via Function Aggregation with Diffusion Models](https://mlsanity.com/paper/2408.08495/) (2024)
- [Overpainting: Localized Context-aware Diffusion Image Editing](https://mlsanity.com/paper/2609.10811/) (2026)
- [CA-Edit: Causality-Aware Condition Adapter for High-Fidelity Local Facial Attribute Editing](https://mlsanity.com/paper/2412.13565/) (2024)
- [UIP2P: Unsupervised Instruction-based Image Editing via Edit Reversibility Constraint](https://mlsanity.com/paper/2412.15216/) (2024)
- [CPO: Condition Preference Optimization for Controllable Image Generation](https://mlsanity.com/paper/2511.04753/) (2025)
- [Chain-of-Image Generation: Toward Monitorable and Controllable Image Generation](https://mlsanity.com/paper/2512.08645/) (2025)
- [Shifting the Breaking Point of Flow Matching for Multi-Instance Editing](https://mlsanity.com/paper/2602.08749/) (2026)
- [Does Synthetic Layered Design Data Benefit Layered Design Decomposition?](https://mlsanity.com/paper/2605.15167/) (2026)
- [SafeCtrl: Region-Based Safety Control for Text-to-Image Diffusion via Detect-Then-Suppress](https://mlsanity.com/paper/2508.11904/) (2025)
- [LAMS-Edit: Latent and Attention Mixing with Schedulers for Improved Content Preservation in Diffusion-Based Image and Style Editing](https://mlsanity.com/paper/2601.02987/) (2026)
- [Localized Latent Editing for Dose-Response Modeling in Botulinum Toxin Injection Planning](https://mlsanity.com/paper/2601.19593/) (2026)
- [FlowBypass: Rectified Flow Trajectory Bypass for Training-Free Image Editing](https://mlsanity.com/paper/2602.01805/) (2026)
- [MIRAGE: Benchmarking and Aligning Multi-Instance Image Editing](https://mlsanity.com/paper/2604.05180/) (2026)
- [Toward 360-Degree Indoor Panorama Editing via Tuning-Free Diffusion Model with Refocusing Cross-Attention](https://mlsanity.com/paper/2606.14035/) (2026)
- [Editing Everything Everywhere All at Once](https://mlsanity.com/paper/2606.31278/) (2026)
- [Owner3D: Ownership-Guided Style Writing for Training-Free Localized 3D Stylization](https://mlsanity.com/paper/2608.14078/) (2026)
- [SR-Edit: Region-Aware Image Editing via Self-Refinement](https://mlsanity.com/paper/2609.02504/) (2026)
- [Attention-Scoped Guidance: Training-Free Spatial Control for Image Editing](https://mlsanity.com/paper/2609.37492/) (2026)

## Abstract (arXiv)

Diffusion models (DMs) have gained prominence due to their ability to generate high-quality varied images with recent advancements in text-to-image generation. The research focus is now shifting towards the controllability of DMs. A significant challenge within this domain is localized editing, where specific areas of an image are modified without affecting the rest of the content. This paper introduces LIME for localized image editing in diffusion models. LIME does not require user-specified regions of interest (RoI) or additional text input, but rather employs features from pre-trained methods and a straightforward clustering method to obtain precise editing mask. Then, by leveraging cross-attention maps, it refines these segments for finding regions to obtain localized edits. Finally, we propose a novel cross-attention regularization technique that penalizes unrelated cross-attention scores in the RoI during the denoising steps, ensuring localized edits. Our approach, without re-training, fine-tuning and additional user inputs, consistently improves the performance of existing methods in various editing benchmarks. The project page can be found at https://enisimsar.github.io/LIME/.