mlsanityData to 7 Oct 2026

Text-Driven Image Editing via Learnable Regions

arXiv 2311.16432 ↗First posted 28 Nov 2023cs.CV cs.AI cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Language has emerged as a natural interface for image editing. In this paper, we introduce a method for region-based image editing driven by textual prompts, without the need for user-provided masks or sketches. Specifically, our approach leverages an existing pre-trained text-to-image model and introduces a bounding box generator to identify the editing regions that are aligned with the textual prompts. We show that this simple approach enables flexible editing that is compatible with current image generation models, and is able to handle complex prompts featuring multiple objects, complex sentences, or lengthy paragraphs. We conduct an extensive user study to compare our method against state-of-the-art methods. The experiments demonstrate the competitive performance of our method in manipulating images with high fidelity and realism that correspond to the provided language descriptions. Our project webpage can be found at: https://yuanze-lin.me/LearnableRegions_page.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 54 of its 61 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

54 of 61 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2015Fast R-CNNcited by 2,930
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  10. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  14. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  15. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  16. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  17. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  18. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  19. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  20. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  21. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  22. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  23. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  24. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  25. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  26. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  27. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  28. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  29. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  30. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  31. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  32. 2022Dual Diffusion Implicit Bridges for Image-to-Image Translationcited by 212
  33. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 211
  34. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  35. 2019Style Transfer by Relaxed Optimal Transport and Self-Similaritycited by 167
  36. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  37. 2021Paint by Wordcited by 107
  38. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 12; 2026: 7.

31 papers.

Sort:

By area:

  1. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  2. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  3. 2024Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editingcited by 34
  4. 2024GoodDrag: Towards Good Practices for Drag Editing with Diffusion Modelscited by 33
  5. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  6. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  7. 2024Image2Struct: Benchmarking Structure Extraction for Vision-Language Modelscited by 18
  8. 2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editingcited by 18
  9. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  10. 2024DreamPolisher: Towards High-Quality Text-to-3D Generation via Geometric Diffusioncited by 12
  11. 2024DiffVax: Optimization-Free Image Immunization Against Diffusion-Based Editingcited by 10
  12. 2025PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Modelscited by 7
19 more
  1. 2025Towards Scalable Human-aligned Benchmark for Text-guided Image Editingcited by 7
  2. 2025Beyond Editing Pairs: Fine-Grained Instructional Image Editing via Multi-Scale Learnable Regionscited by 6
  3. 2025IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generationcited by 6
  4. 2024ParSEL: Parameterized Shape Editing with Languagecited by 5
  5. 2024Olympus: A Universal Task Router for Computer Vision Taskscited by 4
  6. 2025Controllable-Continuous Color Editing in Diffusion Model via Color Mappingcited by 4
  7. 2026TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Modelscited by 4
  8. 2024Mapping the Mind of an Instruction-based Image Editing using SMILEcited by 3
  9. 2025CPAM: Context-Preserving Adaptive Manipulation for Zero-Shot Real Image Editingcited by 2
  10. 2025Immunizing Images from Text to Image Editing via Adversarial Cross-Attentioncited by 2
  11. 2026Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editingcited by 2
  12. 2026Training-free image inversion for one-step diffusion modelscited by 2
  13. 2025Stitch-a-Demo: Video Demonstrations from Multistep Descriptionscited by 1
  14. 2026Overpainting: Localized Context-aware Diffusion Image Editingcited by 1
  15. 2026Generating Chest X-Ray Counterfactuals by Specialising Foundation Image Modelscited by 1
  16. 2025SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integritycited by 0
  17. 2025Region in Context: Text-condition Image editing with Human-like semantic reasoningcited by 0
  18. 2026Localized Control in Diffusion Models via Latent Vector Predictioncited by 0
  19. 2026Geometry-Guided Modeling of Foundation Features Enables Generalizable Object Shape Deformation Learningcited by 0