# Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillation

arXiv: https://arxiv.org/abs/2508.18235 · HTML: https://mlsanity.com/paper/2508.18235/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 37 of its 42 references that are in the map.
- First posted on arXiv 20 Aug 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 37)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Certified Data Removal from Machine Learning Models](https://mlsanity.com/paper/1911.03030/) (2019)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Descent-to-Delete: Gradient-Based Methods for Machine Unlearning](https://mlsanity.com/paper/2007.02923/) (2020)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Unleashing Text-to-Image Diffusion Models for Visual Perception](https://mlsanity.com/paper/2303.02153/) (2023)
- [Rethinking the Backdoor Attacks' Triggers: A Frequency Perspective](https://mlsanity.com/paper/2104.03413/) (2021)
- [Dynamic Backdoor Attacks Against Machine Learning Models](https://mlsanity.com/paper/2003.03675/) (2020)
- [Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing](https://mlsanity.com/paper/2403.03431/) (2024)
- [How to Backdoor Diffusion Models?](https://mlsanity.com/paper/2212.05400/) (2022)
- [Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning](https://mlsanity.com/paper/2305.04175/) (2023)
- [TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets](https://mlsanity.com/paper/2303.05762/) (2023)
- [SODA: Bottleneck Diffusion Models for Representation Learning](https://mlsanity.com/paper/2311.17901/) (2023)
- [Backdooring Convolutional Neural Networks via Targeted Weight Perturbations](https://mlsanity.com/paper/1812.03128/) (2018)
- [Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient](https://mlsanity.com/paper/2405.15304/) (2024)
- [VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models](https://mlsanity.com/paper/2306.06874/) (2023)
- [Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis](https://mlsanity.com/paper/2211.02408/) (2022)
- [Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2407.15861/) (2024)
- [Diffusion Models for Audio Restoration](https://mlsanity.com/paper/2402.09821/) (2024)
- [A Review on Machine Unlearning](https://mlsanity.com/paper/2411.11315/) (2024)
- [Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift](https://mlsanity.com/paper/2312.00050/) (2023)
- [TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors](https://mlsanity.com/paper/2409.05294/) (2024)

## Built on by (most cited first, 2 of 2)

- [Backdoor Sentinel: Detecting and Detoxifying Backdoors in Diffusion Models via Temporal Noise Consistency](https://mlsanity.com/paper/2602.01765/) (2026)
- [PruneForget: Joint Unlearning and Pruning of Vision Models](https://mlsanity.com/paper/2609.32162/) (2026)

## Abstract (arXiv)

Text-to-image diffusion models have revolutionized generative AI, but their vulnerability to backdoor attacks poses significant security risks. Adversaries can inject imperceptible textual triggers into training data, causing models to generate manipulated outputs. Although text-based backdoor defenses in classification models are well-explored, generative models lack effective mitigation techniques against. We address this by selectively erasing the model's learned associations between adversarial text triggers and poisoned outputs, while preserving overall generation quality. Our approach, Self-Knowledge Distillation with Cross-Attention Guidance (SKD-CAG), uses knowledge distillation to guide the model in correcting responses to poisoned prompts while maintaining image quality by exploiting the fact that the backdoored model still produces clean outputs in the absence of triggers. Using the cross-attention mechanism, SKD-CAG neutralizes backdoor influences at the attention level, ensuring the targeted removal of adversarial effects. Extensive experiments show that our method outperforms existing approaches, achieving removal accuracy 100% for pixel backdoors and 93% for style-based attacks, without sacrificing robustness or image fidelity. Our findings highlight targeted unlearning as a promising defense to secure generative models. Code and model weights can be found at https://github.com/Mystic-Slice/Sealing-The-Backdoor .