# Text-to-image Editing by Image Information Removal arXiv: https://arxiv.org/abs/2305.17489 ยท HTML: https://mlsanity.com/paper/2305.17489/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 39 of its 47 references that are in the map. - First posted on arXiv 27 May 2023. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 39) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021) - [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020) - [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022) - [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021) - [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022) - [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018) - [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023) - [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016) - [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017) - [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016) - [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021) - [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019) - [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021) - [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022) - [Splicing ViT Features for Semantic Appearance Transfer](https://mlsanity.com/paper/2201.00424/) (2022) - [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021) - [StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators](https://mlsanity.com/paper/2108.00946/) (2021) - [SINE: SINgle Image Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2212.04489/) (2022) - [ManiGAN: Text-Guided Image Manipulation](https://mlsanity.com/paper/1912.06203/) (2019) ## Built on by (most cited first, 12 of 12) - [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024) - [Improving Tuning-Free Real Image Editing with Proximal Guidance](https://mlsanity.com/paper/2306.05414/) (2023) - [I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing](https://mlsanity.com/paper/2408.14180/) (2024) - [Conditional Image Synthesis with Diffusion Models: A Survey](https://mlsanity.com/paper/2409.19365/) (2024) - [Follow-Your-Shape: Shape-Aware Image Editing via Trajectory-Guided Region Control](https://mlsanity.com/paper/2508.08134/) (2025) - [IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models](https://mlsanity.com/paper/2311.06792/) (2023) - [VINCIE: Unlocking In-context Image Editing from Video](https://mlsanity.com/paper/2506.10941/) (2025) - [FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation](https://mlsanity.com/paper/2405.04834/) (2024) - [A Survey of Multimodal Composite Editing and Retrieval](https://mlsanity.com/paper/2409.05405/) (2024) - [Bridging the Gap Between Multimodal Foundation Models and World Models](https://mlsanity.com/paper/2510.03727/) (2025) - [Gore Diffusion LoRA Model](https://mlsanity.com/paper/2403.08812/) (2024) - [Analyzing Multi-Head Attention on Trojan BERT Models](https://mlsanity.com/paper/2406.16925/) (2024) ## Abstract (arXiv) Diffusion models have demonstrated impressive performance in text-guided image generation. Current methods that leverage the knowledge of these models for image editing either fine-tune them using the input image (e.g., Imagic) or incorporate structure information as additional constraints (e.g., ControlNet). However, fine-tuning large-scale diffusion models on a single image can lead to severe overfitting issues and lengthy inference time. Information leakage from pretrained models also make it challenging to preserve image content not related to the text input. Additionally, methods that incorporate structural guidance (e.g., edge maps, semantic maps, keypoints) find retaining attributes like colors and textures difficult. Using the input image as a control could mitigate these issues, but since these models are trained via reconstruction, a model can simply hide information about the original image when encoding it to perfectly reconstruct the image without learning the editing task. To address these challenges, we propose a text-to-image editing model with an Image Information Removal module (IIR) that selectively erases color-related and texture-related information from the original image, allowing us to better preserve the text-irrelevant content and avoid issues arising from information hiding. Our experiments on CUB, Outdoor Scenes, and COCO reports our approach achieves the best editability-fidelity trade-off results. In addition, a user study on COCO shows that our edited images are preferred 35% more often than prior work.