# Text-Driven Image Editing via Learnable Regions

arXiv: https://arxiv.org/abs/2311.16432 · HTML: https://mlsanity.com/paper/2311.16432/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 61 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 30 of 54)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)

## Built on by (most cited first, 30 of 31)

- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [VHELM: A Holistic Evaluation of Vision Language Models](https://mlsanity.com/paper/2410.07112/) (2024)
- [GoodDrag: Towards Good Practices for Drag Editing with Diffusion Models](https://mlsanity.com/paper/2404.07206/) (2024)
- [Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editing](https://mlsanity.com/paper/2411.15843/) (2024)
- [Conditional Image Synthesis with Diffusion Models: A Survey](https://mlsanity.com/paper/2409.19365/) (2024)
- [IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout](https://mlsanity.com/paper/2506.01949/) (2025)
- [Image2Struct: Benchmarking Structure Extraction for Vision-Language Models](https://mlsanity.com/paper/2410.22456/) (2024)
- [SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing](https://mlsanity.com/paper/2505.02370/) (2025)
- [Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation](https://mlsanity.com/paper/2412.01027/) (2024)
- [DreamPolisher: Towards High-Quality Text-to-3D Generation via Geometric Diffusion](https://mlsanity.com/paper/2403.17237/) (2024)
- [DiffVax: Optimization-Free Image Immunization Against Diffusion-Based Editing](https://mlsanity.com/paper/2411.17957/) (2024)
- [PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Models](https://mlsanity.com/paper/2502.04050/) (2025)
- [Towards Scalable Human-aligned Benchmark for Text-guided Image Editing](https://mlsanity.com/paper/2505.00502/) (2025)
- [Beyond Editing Pairs: Fine-Grained Instructional Image Editing via Multi-Scale Learnable Regions](https://mlsanity.com/paper/2505.19352/) (2025)
- [IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation](https://mlsanity.com/paper/2506.03150/) (2025)
- [ParSEL: Parameterized Shape Editing with Language](https://mlsanity.com/paper/2405.20319/) (2024)
- [Olympus: A Universal Task Router for Computer Vision Tasks](https://mlsanity.com/paper/2412.09612/) (2024)
- [Controllable-Continuous Color Editing in Diffusion Model via Color Mapping](https://mlsanity.com/paper/2509.13756/) (2025)
- [TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Models](https://mlsanity.com/paper/2601.08011/) (2026)
- [Mapping the Mind of an Instruction-based Image Editing using SMILE](https://mlsanity.com/paper/2412.16277/) (2024)
- [CPAM: Context-Preserving Adaptive Manipulation for Zero-Shot Real Image Editing](https://mlsanity.com/paper/2506.18438/) (2025)
- [Immunizing Images from Text to Image Editing via Adversarial Cross-Attention](https://mlsanity.com/paper/2509.10359/) (2025)
- [Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editing](https://mlsanity.com/paper/2603.01535/) (2026)
- [Training-free image inversion for one-step diffusion models](https://mlsanity.com/paper/2606.01380/) (2026)
- [Overpainting: Localized Context-aware Diffusion Image Editing](https://mlsanity.com/paper/2609.10811/) (2026)
- [Stitch-a-Demo: Video Demonstrations from Multistep Descriptions](https://mlsanity.com/paper/2503.13821/) (2025)
- [Generating Chest X-Ray Counterfactuals by Specialising Foundation Image Models](https://mlsanity.com/paper/2609.24879/) (2026)
- [SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity](https://mlsanity.com/paper/2507.23095/) (2025)
- [Region in Context: Text-condition Image editing with Human-like semantic reasoning](https://mlsanity.com/paper/2510.16772/) (2025)
- [Localized Control in Diffusion Models via Latent Vector Prediction](https://mlsanity.com/paper/2602.01991/) (2026)

## Abstract (arXiv)

Language has emerged as a natural interface for image editing. In this paper, we introduce a method for region-based image editing driven by textual prompts, without the need for user-provided masks or sketches. Specifically, our approach leverages an existing pre-trained text-to-image model and introduces a bounding box generator to identify the editing regions that are aligned with the textual prompts. We show that this simple approach enables flexible editing that is compatible with current image generation models, and is able to handle complex prompts featuring multiple objects, complex sentences, or lengthy paragraphs. We conduct an extensive user study to compare our method against state-of-the-art methods. The experiments demonstrate the competitive performance of our method in manipulating images with high fidelity and realism that correspond to the provided language descriptions. Our project webpage can be found at: https://yuanze-lin.me/LearnableRegions_page.