mlsanityData to 14 Oct 2026

DiffEdit: Diffusion-based semantic image editing with mask guidance

arXiv 2210.11427 ↗First posted 20 Oct 2022cs.CVcited by 647

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Image generation has recently seen tremendous advances, with diffusion models allowing to synthesize convincing images for a large variety of text prompts. In this article, we propose DiffEdit, a method to take advantage of text-conditioned diffusion models for the task of semantic image editing, where the goal is to edit an image based on a text query. Semantic image editing is an extension of image generation, with the additional constraint that the generated image should be as similar as possible to a given input image. Current editing methods based on diffusion models usually require to provide a mask, making the task much easier by treating it as a conditional inpainting task. In contrast, our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited, by contrasting predictions of a diffusion model conditioned on different text prompts. Moreover, we rely on latent inference to preserve content in those regions of interest and show excellent synergies with mask-based diffusion. DiffEdit achieves state-of-the-art editing performance on ImageNet. In addition, we evaluate semantic image editing in more challenging settings, using images from the COCO dataset as well as text-based generated images.

Key facts. Cited by 647 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 51 of its 60 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

51 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  2. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  6. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  7. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
32 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  3. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  5. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  6. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  7. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  8. 2020GANSpace: Discovering Interpretable GAN Controlscited by 614
  9. 2018Generative Image Inpainting with Contextual Attentioncited by 580
  10. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  11. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  12. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  13. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  14. 2017Neural Style Transfer: A Reviewcited by 199
  15. 2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
  16. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183
  17. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  18. 2021Paint by Wordcited by 107
  19. 2021GAN-Control: Explicitly Controllable GANscited by 91
  20. 2020Semantic Image Manipulation Using Scene Graphscited by 82
  21. 2018Unsupervised Attention-guided Image to Image Translationcited by 58
  22. 2022Understanding DDPM Latent Codes Through Optimal Transportcited by 45
  23. 2018InstaGAN: Instance-aware Image-to-Image Translationcited by 37
  24. 2022FlexIT: Towards Flexible Semantic Image Translationcited by 28
  25. 2020A Benchmark and Baseline for Language-Driven Image Editingcited by 25
  26. 2018DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)cited by 21
  27. 2021EdiBERT, a generative model for image editingcited by 13
  28. 2022ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generationcited by 13
  29. 2019Evaluating Text-to-Image Matching using Binary Image Selection (BISON)cited by 12
  30. 2020Image-to-Image Translation with Text Guidancecited by 10
  31. 2020Wish You Were Here: Context-Aware Human Generationcited by 10
  32. 2022End-to-End Visual Editing with a Generatively Pre-Trained Artistcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 164; 2024: 201; 2025: 157; 2026: 115.

647 papers.

Sort:

By area:

  1. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  3. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  4. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  5. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  6. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  7. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  8. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  9. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  10. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  11. 2023The Stable Signature: Rooting Watermarks in Latent Diffusion Modelscited by 304
  12. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 301
38 more
  1. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 278
  2. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  3. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 258
  4. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 244
  5. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  6. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  7. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 236
  8. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  9. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  10. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  11. 2023Style Aligned Image Generation via Shared Attentioncited by 216
  12. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  13. 2024Taming Rectified Flow for Inversion and Editingcited by 204
  14. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 201
  15. 2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfercited by 196
  16. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 193
  17. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  18. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  19. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 167
  20. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 165
  21. 2023Dense Text-to-Image Generation with Attention Modulationcited by 163
  22. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  23. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 155
  24. 2023LEDITS++: Limitless Image Editing using Text-to-Image Modelscited by 148
  25. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  26. 2024OmniEdit: Building Image Editing Generalist Models Through Specialist Supervisioncited by 145
  27. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  28. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 142
  29. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  30. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 139
  31. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 137
  32. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 136
  33. 2024Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editingcited by 135
  34. 2024Implicit Style-Content Separation using B-LoRAcited by 135
  35. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
  36. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 130
  37. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  38. 2024ReNoise: Real Image Inversion Through Iterative Noisingcited by 128

50 most cited of 647 All 647 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks
  2. 2024DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing
  3. 2022Blended Latent Diffusion
  4. 2024Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models
  5. 2024An Item is Worth a Prompt: Versatile Image Editing with Disentangled Control
  6. 2025Instructing Text-to-Image Diffusion Models via Classifier-Guided Semantic Optimization
  7. 2022LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models
  8. 2023Textual and Visual Prompt Fusion for Image Editing via Step-Wise Alignment