mlsanityData to 14 Oct 2026

Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models

arXiv 2605.19698 ↗First posted 19 May 2026cs.CR cs.LGcited by 0

On the map: Diffusion models → Robustness, privacy and adversarial aspects

also in 1 more
Abstract (arXiv)

Text-to-image diffusion models are increasingly developed through open-source reuse and repeated downstream fine-tuning, where reused checkpoints are difficult to verify and thus more susceptible to hidden backdoor behaviors. In such ecosystems, a single pretrained model may be sequentially adapted and redistributed by multiple independent parties, allowing multiple concept-specific trigger-target associations to accumulate in the same model. When these associations coexist, semantic conflicts can be amplified in the shared representation space, leading to cross-concept entanglement and degraded generation quality. Notably, instead of strengthening the attack, such accumulation can destabilize previously injected behaviors and reduce attack reliability. In this work, we systematically investigate backdoor attacks under this interference-prone setting and propose Hydra, a unified framework for robust and controlled multi-concept backdoor injection under cumulative and decentralized reuse. Our core insight is that stable backdoor injection under large-scale multi-concept settings requires explicitly constraining trigger semantics while coordinating cross-task interactions during optimization. Specifically, Hydra performs evolutionary trigger search in the text encoder space to identify triggers that are semantically aligned with their target concepts while remaining stable across other injected concepts. It further combines multi-task fine-tuning with trigger-clean regularization to improve training stability under dense multi-concept injection. Extensive experiments across multiple diffusion backbones under rigorous multi-concept settings show that Hydra maintains effective backdoor activation while preserving clean generation fidelity and image quality. For instance, across 8 attackers and 500 concept pairs, Hydra maintains ~95% ASR and strong clean generation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 42 of its 56 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

42 of 56 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Models29 Apr
  2. 2024Understanding Implosion in Text-to-Image Generative Models18 Sep
  3. 2024T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models5 Jul
  4. 2024Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models23 Apr
  5. 2024The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipeline7 Jan
  6. 2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models12 Jun
  7. 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning7 May
  8. 2022How to Backdoor Diffusion Models?11 Dec

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  10. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  12. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
22 more
  1. 2017A Survey on Multi-Task Learningcited by 840
  2. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  3. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  4. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  5. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  6. 2023Effective Data Augmentation With Diffusion Modelscited by 283
  7. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  8. 2023Poisoning Language Models During Instruction Tuningcited by 269
  9. 2024Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editingcited by 135
  10. 2020Graph Optimal Transport for Cross-Domain Alignmentcited by 103
  11. 2023Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Modelscited by 75
  12. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 68
  13. 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesiscited by 59
  14. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 50
  15. 2023Text-image Alignment for Diffusion-based Perceptioncited by 42
  16. 2023BAGM: A Backdoor Attack for Manipulating Text-to-Image Generative Modelscited by 37
  17. 2023Addressing Negative Transfer in Diffusion Modelscited by 33
  18. 2024Diffusion Model with Cross Attention as an Inductive Bias for Disentanglementcited by 27
  19. 2024DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Datacited by 18
  20. 2022Neural Distributed Image Compression with Cross-Attention Feature Alignmentcited by 15
  21. 2022UnGANable: Defending Against GAN-based Face Manipulationcited by 15
  22. 2023Vision Transformer Adapters for Generalizable Multitask Learningcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models
  2. 2026Semantic-level Backdoor Attack against Text-to-Image Diffusion Models
  3. 2026When Backdoors Go Beyond Triggers: Semantic Drift in Diffusion Models Under Encoder Attacks
  4. 2026Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models
  5. 2026Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models
  6. 2026Scaling Exposes the Trigger: Input-Level Backdoor Detection in Text-to-Image Diffusion Models via Cross-Attention Scaling
  7. 2025PEPPER: Perception-Guided Perturbation for Robust Backdoor Defense in Text-to-Image Diffusion Models
  8. 2025Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillation