# Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation

arXiv: https://arxiv.org/abs/2410.15618 · HTML: https://mlsanity.com/paper/2410.15618/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 35 of its 42 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now](https://mlsanity.com/paper/2310.11868/) (2023)
- [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://mlsanity.com/paper/2305.12082/) (2023)
- [Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models](https://mlsanity.com/paper/2305.13873/) (2023)
- [Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts](https://mlsanity.com/paper/2309.06135/) (2023)
- [One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications](https://mlsanity.com/paper/2312.16145/) (2023)
- [Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2405.15234/) (2024)
- [Editing Implicit Assumptions in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.08084/) (2023)
- [Anti-DreamBooth: Protecting users from personalized text-to-image synthesis](https://mlsanity.com/paper/2303.15433/) (2023)
- [EditGuard: Versatile Image Watermarking for Tamper Localization and Copyright Protection](https://mlsanity.com/paper/2312.08883/) (2023)

## Built on by (most cited first, 30 of 50)

- [EraseAnything: Enabling Concept Erasure in Rectified Flow Transformers](https://mlsanity.com/paper/2412.20413/) (2024)
- [Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts](https://mlsanity.com/paper/2504.12782/) (2025)
- [Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them](https://mlsanity.com/paper/2501.18950/) (2025)
- [Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation](https://mlsanity.com/paper/2503.12356/) (2025)
- [One Image is Worth a Thousand Words: A Usability Preservable Text-Image Collaborative Erasing Framework](https://mlsanity.com/paper/2505.11131/) (2025)
- [Minimalist Concept Erasure in Generative Models](https://mlsanity.com/paper/2507.13386/) (2025)
- [Side Effects of Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2508.15124/) (2025)
- [Revoking Amnesia: RL-based Trajectory Optimization to Resurrect Erased Concepts in Diffusion Models](https://mlsanity.com/paper/2510.03302/) (2025)
- [Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppression](https://mlsanity.com/paper/2505.19398/) (2025)
- [Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective](https://mlsanity.com/paper/2511.07970/) (2025)
- [Erased but Not Forgotten: How Backdoors Compromise Concept Erasure](https://mlsanity.com/paper/2504.21072/) (2025)
- [Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.09446/) (2025)
- [SuMa: A Subspace Mapping Approach for Robust and Effective Concept Erasure in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2509.05625/) (2025)
- [Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2510.22851/) (2025)
- [Concept Steerers: Leveraging K-Sparse Autoencoders for Test-Time Controllable Generations](https://mlsanity.com/paper/2501.19066/) (2025)
- [Differential Vector Erasure: Unified Training-Free Concept Erasure for Flow Matching Models](https://mlsanity.com/paper/2602.01089/) (2026)
- [AEGIS: Adversarial Target-Guided Retention-Data-Free Robust Concept Erasure from Diffusion Models](https://mlsanity.com/paper/2602.06771/) (2026)
- [EAR: Erasing Concepts from Unified Autoregressive Models](https://mlsanity.com/paper/2506.20151/) (2025)
- [Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocols](https://mlsanity.com/paper/2507.19894/) (2025)
- [Z-Erase: Enabling Concept Erasure in Single-Stream Diffusion Transformers](https://mlsanity.com/paper/2603.25074/) (2026)
- [ACE: Attentional Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2504.11850/) (2025)
- [Erased, But Not Forgotten: Erased Rectified Flow Transformers Still Remain Unsafe Under Concept Attack](https://mlsanity.com/paper/2510.00635/) (2025)
- [TRACE: Trajectory-Constrained Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2505.23312/) (2025)
- [VCE: Safe Autoregressive Image Generation via Visual Contrast Exploitation](https://mlsanity.com/paper/2509.16986/) (2025)
- [A Unified Framework for Diffusion Model Unlearning with f-Divergence](https://mlsanity.com/paper/2509.21167/) (2025)
- [GrOCE:Graph-Guided Online Concept Erasure for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2511.12968/) (2025)
- [From Unlearning to UNBRANDING: A Benchmark for Trademark-Safe Text-to-Image Generation](https://mlsanity.com/paper/2512.13953/) (2025)
- [MacPrompt: Maraconic-guided Jailbreak against Text-to-Image Models](https://mlsanity.com/paper/2601.07141/) (2026)
- [Beyond Fixed Anchors: Precisely Erasing Concepts with Sibling Exclusive Counterparts](https://mlsanity.com/paper/2510.16342/) (2025)
- [Rethinking Robust Adversarial Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2510.27285/) (2025)

## Abstract (arXiv)

Diffusion models excel at generating visually striking content from text but can inadvertently produce undesirable or harmful content when trained on unfiltered internet data. A practical solution is to selectively removing target concepts from the model, but this may impact the remaining concepts. Prior approaches have tried to balance this by introducing a loss term to preserve neutral content or a regularization term to minimize changes in the model parameters, yet resolving this trade-off remains challenging. In this work, we propose to identify and preserving concepts most affected by parameter changes, termed as textitadversarial concepts. This approach ensures stable erasure with minimal impact on the other concepts. We demonstrate the effectiveness of our method using the Stable Diffusion model, showing that it outperforms state-of-the-art erasure methods in eliminating unwanted content while maintaining the integrity of other unrelated elements. Our code is available at https://github.com/tuananhbui89/Erasing-Adversarial-Preservation.