# Beyond the False Trade-off: Adaptive EWC for Stealthy and Generalizable T2I Backdoors

arXiv: https://arxiv.org/abs/2605.08280 · HTML: https://mlsanity.com/paper/2605.08280/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 23 of 23)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning](https://mlsanity.com/paper/2305.04175/) (2023)
- [TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets](https://mlsanity.com/paper/2303.05762/) (2023)
- [Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models](https://mlsanity.com/paper/2310.13828/) (2023)
- [Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis](https://mlsanity.com/paper/2211.02408/) (2022)
- [The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipeline](https://mlsanity.com/paper/2401.04136/) (2024)
- [Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift](https://mlsanity.com/paper/2312.00050/) (2023)
- [T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models](https://mlsanity.com/paper/2407.04215/) (2024)
- [TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors](https://mlsanity.com/paper/2409.05294/) (2024)
- [UFID: A Unified Framework for Input-level Backdoor Detection on Diffusion Models](https://mlsanity.com/paper/2404.01101/) (2024)
- [Diff-Cleanse: Identifying and Mitigating Backdoor Attacks in Diffusion Models](https://mlsanity.com/paper/2407.21316/) (2024)
- [Backdooring Bias (B2) into Stable Diffusion Models](https://mlsanity.com/paper/2406.15213/) (2024)
- [Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.09669/) (2025)
- [UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Models](https://mlsanity.com/paper/2412.11441/) (2024)
- [Efficient Input-level Backdoor Defense on Text-to-Image Synthesis via Neuron Activation Variation](https://mlsanity.com/paper/2503.06453/) (2025)
- [Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2504.20518/) (2025)
- [BitAbuse: A Dataset of Visually Perturbed Texts for Defending Phishing Attacks](https://mlsanity.com/paper/2502.05225/) (2025)

## Abstract (arXiv)

Preserving model fidelity is essential for stealthy text-to-image (T2I) backdoor attacks. Existing methods such as Learning without Forgetting (LwF) rely on output-based distillation, which provides limited regularization. We introduce Elastic Weight Consolidation (EWC) as a parameter-based alternative for preserving fidelity in backdoor learning. While stronger in principle, we show that standard static EWC with a fixed regularization weight lambda and mean-squared utility loss creates an artificial trade-off between attack success rate (ASR) and fidelity, particularly degrading performance on weak triggers. To address this, we propose Cosine-Aware Adaptive EWC, which dynamically adjusts EWC regularization using a cosine-based semantic utility and adaptive scheduling. This approach transforms EWC from a fixed penalty into a context-sensitive constraint, maintaining high ASR while preserving model fidelity. Experiments demonstrate improved ASR-fidelity balance and enhanced robustness on out-of-domain (OOD) datasets compared to existing baselines.