# Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimization

arXiv: https://arxiv.org/abs/2504.09039 · HTML: https://mlsanity.com/paper/2504.09039/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 12 Apr 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Certified Data Removal from Machine Learning Models](https://mlsanity.com/paper/1911.03030/) (2019)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [A Survey of Machine Unlearning](https://mlsanity.com/paper/2209.02299/) (2022)
- [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization](https://mlsanity.com/paper/1902.05967/) (2019)
- [fastai: A Layered API for Deep Learning](https://mlsanity.com/paper/2002.04688/) (2020)
- [Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Models](https://mlsanity.com/paper/2305.10120/) (2023)

## Built on by (most cited first, 19 of 19)

- [Mass Concept Erasure in Diffusion Models with Concept Hierarchy](https://mlsanity.com/paper/2601.03305/) (2026)
- [SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders](https://mlsanity.com/paper/2509.21379/) (2025)
- [Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective](https://mlsanity.com/paper/2511.07970/) (2025)
- [EraseFlow: Learning Concept Erasure Policies via GFlowNet-Driven Alignment](https://mlsanity.com/paper/2511.00804/) (2025)
- [Forget Many, Forget Right: Scalable and Precise Concept Unlearning in Diffusion Models](https://mlsanity.com/paper/2601.06162/) (2026)
- [Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks](https://mlsanity.com/paper/2607.07907/) (2026)
- [DyME: Dynamic Multi-Concept Erasure in Diffusion Models with Bi-Level Orthogonal LoRA Adaptation](https://mlsanity.com/paper/2509.21433/) (2025)
- [Locality-Aware Continual Unlearning for Diffusion Models](https://mlsanity.com/paper/2512.02657/) (2025)
- [Forget-It-All: Multi-Concept Machine Unlearning via Concept-Aware Neuron Masking](https://mlsanity.com/paper/2601.06163/) (2026)
- [UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning](https://mlsanity.com/paper/2602.03410/) (2026)
- [GrOCE:Graph-Guided Online Concept Erasure for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2511.12968/) (2025)
- [Roots Beneath the Cut: Uncovering the Risk of Concept Revival in Pruning-Based Unlearning for Diffusion Models](https://mlsanity.com/paper/2603.06640/) (2026)
- [Prototype-Guided Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2603.08271/) (2026)
- [FADE: Frame-Aware Diffusion-Transformer-based Multi-Concept Erasure for Video Unlearning](https://mlsanity.com/paper/2610.03980/) (2026)
- [No Concept Escapes the Audit: Auditing-Aware Unlearning for Verifiable Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2610.05401/) (2026)
- [Your Unlearning Gives You Away: Identifying Erased Concepts in Diffusion Models](https://mlsanity.com/paper/2610.05601/) (2026)
- [Mitigating Sequential Reappearance in Diffusion Data-Point Unlearning](https://mlsanity.com/paper/2609.25166/) (2026)
- [Continual Concept Erasure in Diffusion Models by Suppressing Cross-Edit Interference](https://mlsanity.com/paper/2610.01989/) (2026)
- [Compositional Concepts Erasing: Disentangling Object-Style via Domain Manifold Unlearning](https://mlsanity.com/paper/2610.14514/) (2026)

## Abstract (arXiv)

Text-to-image (T2I) diffusion models have achieved remarkable success in generating high-quality images from textual prompts. However, their ability to store vast amounts of knowledge raises concerns in scenarios where selective forgetting is necessary, such as removing copyrighted content, reducing biases, or eliminating harmful concepts. While existing unlearning methods can remove certain concepts, they struggle with multi-concept forgetting due to instability, residual knowledge persistence, and generation quality degradation. To address these challenges, we propose Dynamic Mask coupled with Concept-Aware Loss, a novel unlearning framework designed for multi-concept forgetting in diffusion models. Our Dynamic Mask mechanism adaptively updates gradient masks based on current optimization states, allowing selective weight modifications that prevent interference with unrelated knowledge. Additionally, our Concept-Aware Loss explicitly guides the unlearning process by enforcing semantic consistency through superclass alignment, while a regularization loss based on knowledge distillation ensures that previously unlearned concepts remain forgotten during sequential unlearning. We conduct extensive experiments to evaluate our approach. Results demonstrate that our method outperforms existing unlearning techniques in forgetting effectiveness, output fidelity, and semantic coherence, particularly in multi-concept scenarios. Our work provides a principled and flexible framework for stable and high-fidelity unlearning in generative models. The code will be released publicly.