# Addressing Text Embedding Leakage in Diffusion-based Image Editing

arXiv: https://arxiv.org/abs/2412.04715 · HTML: https://mlsanity.com/paper/2412.04715/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting](https://mlsanity.com/paper/2212.06909/) (2022)
- [Splicing ViT Features for Semantic Appearance Transfer](https://mlsanity.com/paper/2201.00424/) (2022)
- [Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment](https://mlsanity.com/paper/2306.08877/) (2023)
- [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://mlsanity.com/paper/2310.01506/) (2023)
- [Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing](https://mlsanity.com/paper/2403.03431/) (2024)
- [Inversion-Free Image Editing with Natural Language](https://mlsanity.com/paper/2312.04965/) (2023)
- [Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models](https://mlsanity.com/paper/2305.04441/) (2023)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing](https://mlsanity.com/paper/2309.15664/) (2023)
- [StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing](https://mlsanity.com/paper/2303.15649/) (2023)
- [Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation](https://mlsanity.com/paper/2312.10113/) (2023)
- [Zero-shot Referring Image Segmentation with Global-Local Context Features](https://mlsanity.com/paper/2303.17811/) (2023)
- [Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis](https://mlsanity.com/paper/2411.07132/) (2024)
- [Direct Inversion: Optimization-Free Text-Driven Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.07825/) (2022)

## Built on by (most cited first, 7 of 7)

- [I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing](https://mlsanity.com/paper/2601.03741/) (2026)
- [Shifting the Breaking Point of Flow Matching for Multi-Instance Editing](https://mlsanity.com/paper/2602.08749/) (2026)
- [T2LSC-Bench: Benchmarking Localized Semantic Control in Text-to-Image Generation](https://mlsanity.com/paper/2609.02255/) (2026)
- [The Attention Triangle in Audio-Video Models](https://mlsanity.com/paper/2609.03586/) (2026)
- [Editing Everything Everywhere All at Once](https://mlsanity.com/paper/2606.31278/) (2026)
- [Localize, Don't Beautify: Client-Side Control of Image-Editing APIs for Cosmetic Surgery Previews](https://mlsanity.com/paper/2608.02841/) (2026)
- [Measured Sliders: Learning Continuous Controls from Differentiable Image Measurements](https://mlsanity.com/paper/2609.05234/) (2026)

## Abstract (arXiv)

Text-based image editing, powered by generative diffusion models, lets users modify images through natural-language prompts and has dramatically simplified traditional workflows. Despite these advances, current methods still suffer from a critical problem: attribute leakage, where edits meant for specific objects unintentionally affect unrelated regions or other target objects. Our analysis reveals the root cause as the semantic entanglement inherent in End-of-Sequence (EOS) embeddings generated by autoregressive text encoders, which indiscriminately aggregate attributes across prompts. To address this issue, we introduce Attribute-Leakage-free Editing (ALE), a framework that tackles attribute leakage at its source. ALE combines Object-Restricted Embeddings (ORE) to disentangle text embeddings, Region-Guided Blending for Cross-Attention Masking (RGB-CAM) for spatially precise attention, and Background Blending (BB) to preserve non-edited content. To quantitatively evaluate attribute leakage across various editing methods, we propose the Attribute-Leakage Evaluation Benchmark (ALE-Bench), featuring comprehensive editing scenarios and new metrics. Extensive experiments show that ALE reduces attribute leakage by large margins, thereby enabling accurate, multi-object, text-driven image editing while faithfully preserving non-target content.