mlsanityData to 14 Oct 2026

Blended Diffusion for Text-driven Editing of Natural Images

arXiv 2111.14818 ↗First posted 29 Nov 2021cs.CV cs.GR cs.LGcited by 927

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Natural language offers a highly intuitive interface for image editing. In this paper, we introduce the first solution for performing local (region-based) edits in generic natural images, based on a natural language description along with an ROI mask. We achieve our goal by leveraging and combining a pretrained language-image model (CLIP), to steer the edit towards a user-provided text prompt, with a denoising diffusion probabilistic model (DDPM) to generate natural-looking results. To seamlessly fuse the edited region with the unchanged parts of the image, we spatially blend noised versions of the input image with the local text-guided diffusion latent at a progression of noise levels. In addition, we show that adding augmentations to the diffusion process mitigates adversarial results. We compare against several baselines and related methods, both qualitatively and quantitatively, and show that our method outperforms these solutions in terms of overall realism, ability to preserve the background and matching the text. Finally, we show several text-driven editing applications, including adding a new object to an image, removing/replacing/altering existing objects, background replacement, and image extrapolation. Code is available at: https://omriavrahami.com/blended-diffusion-page/

Key facts. Cited by 927 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 39 of its 63 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

39 of 63 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  12. 2013Intriguing properties of neural networkscited by 5,155
27 more
  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  8. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  9. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  10. 2016Generative Adversarial Text to Image Synthesiscited by 735
  11. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  13. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  14. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  15. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
  16. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  17. 2021GAN Inversion: A Surveycited by 367
  18. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  19. 2021ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinementcited by 248
  20. 2019Controllable Text-to-Image Generationcited by 187
  21. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  22. 2015Generating Images from Captions with Attentioncited by 136
  23. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  24. 2021Paint by Wordcited by 107
  25. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  26. 2018Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Networkcited by 64
  27. 2021Towards Open-World Text-Guided Face Image Generation and Manipulationcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 71; 2023: 277; 2024: 300; 2025: 181; 2026: 95.

927 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  8. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  9. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  10. 2022Diffusion Models in Vision: A Surveycited by 1,015
  11. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  12. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
38 more
  1. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  2. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 647
  3. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  4. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  5. 2023Zero-shot Image-to-Image Translationcited by 505
  6. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  7. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 489
  8. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  9. 2022Blended Latent Diffusioncited by 432
  10. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 412
  11. 2023DIRE for Diffusion-Generated Image Detectioncited by 390
  12. 2023ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shiftingcited by 365
  13. 2023DeepCache: Accelerating Diffusion Models for Freecited by 359
  14. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  15. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  16. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 340
  17. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  18. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  19. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 324
  20. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  21. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 294
  22. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 288
  23. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  24. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 278
  25. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  26. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  27. 2023DiffBIR: Towards Blind Image Restoration with Generative Diffusion Priorcited by 257
  28. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  29. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 244
  30. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  31. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  32. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  33. 2023TextDiffuser: Diffusion Models as Text Painterscited by 229
  34. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  35. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 212
  36. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 212
  37. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  38. 2023A Survey on Video Diffusion Modelscited by 206

50 most cited of 927 All 927 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Real-time Text-driven Image Manipulation with Unconditional Diffusion Models
  2. 2023Custom-Edit: Text-Guided Image Editing with Customized Diffusion Models
  3. 2025PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Models
  4. 2022LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models
  5. 2026Language-Free Generative Editing from One Visual Example
  6. 2023EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods
  7. 2024Task-Oriented Diffusion Inversion for High-Fidelity Text-based Editing
  8. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoders