mlsanityData to 7 Oct 2026

Blended Latent Diffusion

arXiv 2206.02779 ↗First posted 6 Jun 2022cs.CV cs.GR cs.LGcited by 430

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

also in 1 more
Abstract (arXiv)

The tremendous progress in neural image generation, coupled with the emergence of seemingly omnipotent vision-language models has finally enabled text-based interfaces for creating and editing images. Handling generic images requires a diverse underlying generative model, hence the latest works utilize diffusion models, which were shown to surpass GANs in terms of diversity. One major drawback of diffusion models, however, is their relatively slow inference time. In this paper, we present an accelerated solution to the task of local text-driven editing of generic images, where the desired edits are confined to a user-provided mask. Our solution leverages a recent text-to-image Latent Diffusion Model (LDM), which speeds up diffusion by operating in a lower-dimensional latent space. We first convert the LDM into a local image editor by incorporating Blended Diffusion into it. Next we propose an optimization-based solution for the inherent inability of this LDM to accurately reconstruct images. Finally, we address the scenario of performing local edits using thin masks. We evaluate our method against the available baselines both qualitatively and quantitatively and demonstrate that in addition to being faster, our method achieves better precision than the baselines while mitigating some of their artifacts.

Key facts. Cited by 430 papers in the map (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 86 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

56 of 86 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
38 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  13. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  14. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  15. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  16. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  17. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  18. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  19. 2016Generative Adversarial Text to Image Synthesiscited by 735
  20. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  21. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  22. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  23. 2021Score-based Generative Modeling in Latent Spacecited by 634
  24. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  25. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  26. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  27. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  28. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  29. 2021GAN Inversion: A Surveycited by 366
  30. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  31. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  32. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  33. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  34. 2020Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulationcited by 225
  35. 2020Semantic Photo Manipulation with a Generative Image Priorcited by 178
  36. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  37. 2015Generating Images from Captions with Attentioncited by 135
  38. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 24; 2023: 109; 2024: 135; 2025: 109; 2026: 53.

430 papers.

Sort:

By area:

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  5. 2022Diffusion Models in Vision: A Surveycited by 1,008
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  7. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  8. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  9. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  10. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  11. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  12. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
38 more
  1. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  2. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  3. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  4. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  5. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  6. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  7. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  8. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  9. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  10. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  11. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  12. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  13. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  14. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  15. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  16. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  17. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 200
  18. 2024Taming Rectified Flow for Inversion and Editingcited by 200
  19. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  20. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  21. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  22. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  23. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  24. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  25. 2023LEDITS++: Limitless Image Editing using Text-to-Image Modelscited by 147
  26. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  27. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  28. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 135
  29. 2024Implicit Style-Content Separation using B-LoRAcited by 135
  30. 2022TruFor: Leveraging all-round clues for trustworthy image forgery detection and localizationcited by 130
  31. 2024ReNoise: Real Image Inversion Through Iterative Noisingcited by 127
  32. 2023Delta Denoising Scorecited by 120
  33. 2022Uncovering the Disentanglement Capability in Text-to-Image Diffusion Modelscited by 116
  34. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Modelscited by 113
  35. 2024Stable Flow: Vital Layers for Training-Free Image Editingcited by 111
  36. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusionscited by 108
  37. 2024Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advancescited by 105
  38. 2023DiffiT: Diffusion Vision Transformers for Image Generationcited by 104

50 most cited of 430 All 430 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enabling Local Editing in Diffusion Models by Joint and Individual Component Analysis
  2. 2023Text-driven Visual Synthesis with Latent Diffusion Prior
  3. 2022LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models
  4. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance
  5. 2023Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images
  6. 2023Forgedit: Text Guided Image Editing via Learning and Forgetting
  7. 2023PRedItOR: Text Guided Image Editing with Diffusion Prior
  8. 2023Filter-Guided Diffusion for Controllable Image Generation