# Erased but Not Forgotten: How Backdoors Compromise Concept Erasure

arXiv: https://arxiv.org/abs/2504.21072 · HTML: https://mlsanity.com/paper/2504.21072/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 79 of its 109 references that are in the map.
- First posted on arXiv 29 Apr 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 30 of 79)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)

## Built on by (most cited first, 9 of 9)

- [Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning](https://mlsanity.com/paper/2510.17021/) (2025)
- [Memory Self-Regeneration: Uncovering Hidden Knowledge in Unlearned Models](https://mlsanity.com/paper/2510.03263/) (2025)
- [From Unlearning to UNBRANDING: A Benchmark for Trademark-Safe Text-to-Image Generation](https://mlsanity.com/paper/2512.13953/) (2025)
- [GEM: Geometric Erasure by Contrastive Velocity Matching in Rectified Flows](https://mlsanity.com/paper/2606.00140/) (2026)
- [Steerability of Instrumental-Convergence Tendencies in LLMs](https://mlsanity.com/paper/2601.01584/) (2026)
- [Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models](https://mlsanity.com/paper/2605.19227/) (2026)
- [Obliviate: Erasing Concepts from Autoregressive Image Generation Models](https://mlsanity.com/paper/2606.28643/) (2026)
- [Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models](https://mlsanity.com/paper/2603.04064/) (2026)
- [Whispers in the Noise: Surrogate-Guided Concept Awakening via a Multi-Agent Framework](https://mlsanity.com/paper/2605.18150/) (2026)

## Abstract (arXiv)

The expansion of text-to-image diffusion models has raised concerns about harmful outputs, from fabricated depictions of public figures to sexually explicit imagery. To mitigate such risks, prior work has proposed concept erasure methods that aim to sever unwanted concepts from the model via fine-tuning, yet it remains unclear whether these approaches truly remove all links to the harmful concept or merely conceal superficial connections. In this work, we reveal a critical vulnerability, the Erasure Evasion Backdoor (EEB): an adversary binds a backdoor trigger to a concept slated for removal, and this malicious link survives subsequent erasure. We show that both black-box and white-box adversaries can instantiate this threat. Across six state-of-the-art erasure methods, including robust ones that explicitly search for alternative representations of the target concept, EEB consistently exposes harmful content: up to 82% success against celebrity-identity unlearning, up to 94% for object erasure, and up to 16 times amplification of explicit-content exposure. While EEB uncovers a blind spot in current erasure methods, it also provides a diagnostic tool for stress-testing future concept erasure techniques. Our code is available at https://github.com/multimodal-ai-lab/EEB.