# Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective arXiv: https://arxiv.org/abs/2511.07970 ยท HTML: https://mlsanity.com/paper/2511.07970/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. - Builds on 55 of its 64 references that are in the map. - First posted on arXiv 11 Nov 2025. ## Placed in - Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/) - LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/) ## Builds on (most cited first, 30 of 55) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022) - [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021) - [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017) - [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022) - [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022) - [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017) - [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022) - [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017) - [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018) - [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023) - [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023) - [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018) - [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022) - [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023) - [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019) - [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021) - [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021) - [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021) - [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023) - [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022) - [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020) - [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020) - [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022) - [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023) - [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023) - [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024) ## Built on by (most cited first, 10 of 10) - [OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasure](https://mlsanity.com/paper/2603.11493/) (2026) - [Locality-Aware Continual Unlearning for Diffusion Models](https://mlsanity.com/paper/2512.02657/) (2025) - [Lessons and Open Questions from a Unified Study of Camera-Trap Species Recognition Over Time](https://mlsanity.com/paper/2603.20509/) (2026) - [Continual Speaker Identity Unlearning with Minimal Interference](https://mlsanity.com/paper/2605.25962/) (2026) - [Modular Energy Steering for Safe Text-to-Image Generation with Foundation Models](https://mlsanity.com/paper/2604.02265/) (2026) - [Mitigating Sequential Reappearance in Diffusion Data-Point Unlearning](https://mlsanity.com/paper/2609.25166/) (2026) - [Continual Data Unlearning in Diffusion Models via Transition-based Regularization](https://mlsanity.com/paper/2609.32328/) (2026) - [Motion Concept Unlearning in Video Diffusion Models](https://mlsanity.com/paper/2609.36832/) (2026) - [Continual Concept Erasure in Diffusion Models by Suppressing Cross-Edit Interference](https://mlsanity.com/paper/2610.01989/) (2026) - [Compositional Concepts Erasing: Disentangling Object-Style via Domain Manifold Unlearning](https://mlsanity.com/paper/2610.14514/) (2026) ## Abstract (arXiv) Machine unlearning--the ability to remove designated concepts from a pre-trained model--has advanced rapidly, particularly for text-to-image diffusion models. However, existing methods typically assume that unlearning requests arrive all at once, whereas in practice they often arrive sequentially. We present the first systematic study of continual unlearning in text-to-image diffusion models and show that popular unlearning methods suffer from rapid utility collapse: after only a few requests, models forget retained knowledge and generate degraded images. We trace this failure to cumulative parameter drift from the pre-training weights and argue that regularization is crucial to addressing it. To this end, we study a suite of add-on regularizers that (1) mitigate drift and (2) remain compatible with existing unlearning methods. Beyond generic regularizers, we show that semantic awareness is essential for preserving concepts close to the unlearning target, and propose a gradient-projection method that constrains parameter drift orthogonal to their subspace. This substantially improves continual unlearning performance and is complementary to other regularizers for further gains. Taken together, our study establishes continual unlearning as a fundamental challenge in text-to-image generation and provides insights, baselines, and open directions for advancing safe and accountable generative AI.