mlsanityData to 7 Oct 2026

Ablating Concepts in Text-to-Image Diffusion Models

arXiv 2303.13516 ↗First posted 23 Mar 2023cs.CV cs.GR cs.LGcited by 312

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

Large-scale text-to-image diffusion models can generate high-fidelity images with powerful compositional ability. However, these models are typically trained on an enormous amount of Internet data, often containing copyrighted material, licensed images, and personal photos. Furthermore, they have been found to replicate the style of various living artists or memorize exact training samples. How can we remove such copyrighted concepts or images without retraining the model from scratch? To achieve this goal, we propose an efficient method of ablating concepts in the pretrained model, i.e., preventing the generation of a target concept. Our algorithm learns to match the image distribution for a target style, instance, or text prompt we wish to ablate to the distribution corresponding to an anchor concept. This prevents the model from generating target concepts given its text condition. Extensive experiments show that our method can successfully prevent the generation of the ablated concept while preserving closely related concepts in the model.

Key facts. Cited by 312 papers in the map (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 71 of its 84 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

71 of 84 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Erasing Concepts from Diffusion Models13 Mar
  2. 2023Extracting Training Data from Diffusion Models30 Jan
  3. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models9 Nov

Other references · 68

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  12. 2022Locating and Editing Factual Associations in GPTcited by 2,609
38 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  9. 2018Demystifying MMD GANscited by 1,268
  10. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  12. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  13. 2019Machine Unlearningcited by 1,020
  14. 2022Mass-Editing Memory in a Transformercited by 976
  15. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  16. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  17. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  18. 2016Generative Adversarial Text to Image Synthesiscited by 735
  19. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  20. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  21. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  22. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  23. 2023Zero-shot Image-to-Image Translationcited by 503
  24. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  25. 2021Fast Model Editing at Scalecited by 491
  26. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  27. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  28. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  29. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  30. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 360
  31. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  32. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  33. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  34. 2020Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulationcited by 225
  35. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  36. 2021Few-shot Image Generation via Cross-domain Correspondencecited by 206
  37. 2023Glaze: Protecting Artists from Style Mimicry by Text-to-Image Modelscited by 200
  38. 2020Semantic Photo Manipulation with a Generative Image Priorcited by 178

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 38; 2024: 95; 2025: 106; 2026: 73.

312 papers.

Sort:

By area:

  1. 2023Unified Concept Editing in Diffusion Modelscited by 377
  2. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 298
  3. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  4. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  5. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 239
  6. 2023Understanding and Mitigating Copying in Diffusion Modelscited by 205
  7. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  8. 2023MMA-Diffusion: MultiModal Attack on Diffusion Modelscited by 184
  9. 2023One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applicationscited by 164
  10. 2024Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Modelscited by 164
  11. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  12. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
38 more
  1. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  2. 2023Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Modelscited by 148
  3. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  4. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  5. 2023Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Eraserscited by 118
  6. 2024EraseAnything: Enabling Concept Erasure in Rectified Flow Transformerscited by 93
  7. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  8. 2023Circumventing Concept Erasure Methods For Text-to-Image Generative Modelscited by 81
  9. 2023Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editingcited by 76
  10. 2025SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoderscited by 75
  11. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  12. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  13. 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 68
  14. 2023Toward effective protection against diffusion based mimicry through score distillationcited by 64
  15. 2024Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Modelscited by 62
  16. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 61
  17. 2025Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Conceptscited by 61
  18. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  19. 2024R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Modelcited by 56
  20. 2024Erasing Undesirable Influence in Diffusion Modelscited by 52
  21. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 51
  22. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  23. 2024Towards Memorization-Free Diffusion Modelscited by 49
  24. 2024Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservationcited by 48
  25. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  26. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
  27. 2024Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancementcited by 43
  28. 2023Towards Safe Self-Distillation of Internet-Scale Text-to-Image Diffusion Modelscited by 41
  29. 2024STEREO: A Two-Stage Framework for Adversarially Robust Concept Erasing from Text-to-Image Diffusion Modelscited by 39
  30. 2024Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Conceptscited by 39
  31. 2024Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matterscited by 38
  32. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 35
  33. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  34. 2024Erasing Conceptual Knowledge from Language Modelscited by 34
  35. 2025Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Themcited by 34
  36. 2023Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inferencecited by 33
  37. 2023All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Modelscited by 33
  38. 2024The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipelinecited by 33

50 most cited of 312 All 312 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Unlearning Concepts from Text-to-Video Diffusion Models
  2. 2025Concept-TRAK: Understanding how diffusion models learn concepts through concept-level attribution
  3. 2026Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models
  4. 2023Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion
  5. 2023The Hidden Language of Diffusion Models
  6. 2025ACE: Attentional Concept Erasure in Diffusion Models
  7. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Models
  8. 2024Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models