# Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearning

arXiv: https://arxiv.org/abs/2405.07288 · HTML: https://mlsanity.com/paper/2405.07288/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 57 of its 72 references that are in the map.
- First posted on arXiv 12 May 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 30 of 57)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Fast Sampling of Diffusion Models with Exponential Integrator](https://mlsanity.com/paper/2204.13902/) (2022)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)

## Built on by (most cited first, 29 of 29)

- [Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety](https://mlsanity.com/paper/2502.05206/) (2025)
- [Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts](https://mlsanity.com/paper/2504.12782/) (2025)
- [Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey](https://mlsanity.com/paper/2408.03400/) (2024)
- [When Machine Unlearning Meets Retrieval-Augmented Generation (RAG): Keep Secret or Forget Knowledge?](https://mlsanity.com/paper/2410.15267/) (2024)
- [CLIPErase: Efficient Unlearning of Visual-Textual Associations in CLIP](https://mlsanity.com/paper/2410.23330/) (2024)
- [MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning](https://mlsanity.com/paper/2406.14796/) (2024)
- [Unstable Unlearning: The Hidden Risk of Concept Resurgence in Diffusion Models](https://mlsanity.com/paper/2410.08074/) (2024)
- [VideoEraser: Concept Erasure in Text-to-Video Diffusion Models](https://mlsanity.com/paper/2508.15314/) (2025)
- [Unlearning or Concealment? A Critical Analysis and Evaluation Metrics for Unlearning in Diffusion Models](https://mlsanity.com/paper/2409.05668/) (2024)
- [Text Guided Image Editing with Automatic Concept Locating and Forgetting](https://mlsanity.com/paper/2405.19708/) (2024)
- [Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Concepts](https://mlsanity.com/paper/2501.09833/) (2025)
- [Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppression](https://mlsanity.com/paper/2505.19398/) (2025)
- [MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering](https://mlsanity.com/paper/2510.04217/) (2025)
- [Unlearning Concepts from Text-to-Video Diffusion Models](https://mlsanity.com/paper/2407.14209/) (2024)
- [OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasure](https://mlsanity.com/paper/2603.11493/) (2026)
- [How Data Inter-connectivity Shapes LLMs Unlearning: A Structural Unlearning Perspective](https://mlsanity.com/paper/2406.16810/) (2024)
- [Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting](https://mlsanity.com/paper/2411.16769/) (2024)
- [SecureT2I: No More Unauthorized Manipulation on AI Generated Images from Prompts](https://mlsanity.com/paper/2507.03636/) (2025)
- [Beyond Fixed Anchors: Precisely Erasing Concepts with Sibling Exclusive Counterparts](https://mlsanity.com/paper/2510.16342/) (2025)
- [Rethinking Robust Adversarial Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2510.27285/) (2025)
- [Coffee: Controllable Diffusion Fine-tuning](https://mlsanity.com/paper/2511.14113/) (2025)
- [Evaluation without Generation: Non-Generative Assessment of Harmful Model Specialization with Applications to CSAM](https://mlsanity.com/paper/2604.25119/) (2026)
- [P2-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization](https://mlsanity.com/paper/2606.03376/) (2026)
- [Semantic Steering for Controllable Generation: Tuning-Free Concept Erasure in Multimodal Diffusion Transformers](https://mlsanity.com/paper/2608.12829/) (2026)
- [FameBias: Embedding Manipulation Bias Attack in Text-to-Image Models](https://mlsanity.com/paper/2412.18302/) (2024)
- [Few-Shot Concept Unlearning with Low Rank Adaptation](https://mlsanity.com/paper/2505.12395/) (2025)
- [Why Instruction-Based Unlearning Fails in Diffusion Models?](https://mlsanity.com/paper/2604.01514/) (2026)
- [CoreUnlearn: Rethinking Concept Unlearning through Disentangled Component-Level Erasure in Text-guided Diffusion Models](https://mlsanity.com/paper/2606.01658/) (2026)
- [A Survey on Adversarial Attacks and Defenses for Diffusion Models Across Multiple Modalities](https://mlsanity.com/paper/2609.05503/) (2026)

## Abstract (arXiv)

Generating images from text has become easier because of the scaling of diffusion models and advancements in the field of vision and language. These models are trained using vast amounts of data from the Internet. Hence, they often contain undesirable content such as copyrighted material. As it is challenging to remove such data and retrain the models, methods for erasing specific concepts from pre-trained models have been investigated. We propose a novel concept-erasure method that updates the text encoder using few-shot unlearning in which a few real images are used. The discussion regarding the generated images after erasing a concept has been lacking. While there are methods for specifying the transition destination for concepts, the validity of the specified concepts is unclear. Our method implicitly achieves this by transitioning to the latent concepts inherent in the model or the images. Our method can erase a concept within 10 s, making concept erasure more accessible than ever before. Implicitly transitioning to related concepts leads to more natural concept erasure. We applied the proposed method to various concepts and confirmed that concept erasure can be achieved tens to hundreds of times faster than with current methods. By varying the parameters to be updated, we obtained results suggesting that, like previous research, knowledge is primarily accumulated in the feed-forward networks of the text encoder. Our code is available at urlhttps://github.com/fmp453/few-shot-erasing