# Few-Shot Concept Unlearning with Low Rank Adaptation

arXiv: https://arxiv.org/abs/2505.12395 · HTML: https://mlsanity.com/paper/2505.12395/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 9 of 9)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Scissorhands: Scrub Data Influence via Connection Sensitivity in Networks](https://mlsanity.com/paper/2401.06187/) (2024)
- [UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models](https://mlsanity.com/paper/2402.11846/) (2024)
- [Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearning](https://mlsanity.com/paper/2405.07288/) (2024)

## Abstract (arXiv)

Image Generation models are a trending topic nowadays, with many people utilizing Artificial Intelligence models in order to generate images. There are many such models which, given a prompt of a text, will generate an image which depicts said prompt. There are many image generation models, such as Latent Diffusion Models, Denoising Diffusion Probabilistic Models, Generative Adversarial Networks and many more. When generating images, these models can generate sensitive image data, which can be threatening to privacy or may violate copyright laws of private entities. Machine unlearning aims at removing the influence of specific data subsets from the trained models and in the case of image generation models, remove the influence of a concept such that the model is unable to generate said images of the concept when prompted. Conventional retraining of the model can take upto days, hence fast algorithms are the need of the hour. In this paper we propose an algorithm that aims to remove the influence of concepts in diffusion models through updating the gradients of the final layers of the text encoders. Using a weighted loss function, we utilize backpropagation in order to update the weights of the final layers of the Text Encoder componet of the Stable Diffusion Model, removing influence of the concept from the text-image embedding space, such that when prompted, the result is an image not containing the concept. The weighted loss function makes use of Textual Inversion and Low-Rank Adaptation.We perform our experiments on Latent Diffusion Models, namely the Stable Diffusion v2 model, with an average concept unlearning runtime of 50 seconds using 4-5 images.