When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models
On the map: Diffusion models → Robustness, privacy and adversarial aspects
also in 1 more
Abstract (arXiv)
While diffusion models have revolutionized visual content generation, their rapid adoption has underscored the critical need to investigate vulnerabilities, e.g., to backdoor attacks. In multimodal diffusion models, it is natural to expect that attacking multiple modalities simultaneously (e.g., text and image) would yield complementary effects and strengthen the overall backdoor. In this paper, we challenge this assumption by investigating the phenomenon of Backdoor Modality Collapse, a scenario where the backdoor mechanism degenerates to rely predominantly on a subset of modalities, rendering others redundant. To rigorously quantify this behavior, we introduce two novel metrics: Trigger Modality Attribution (TMA) and Cross-Trigger Interaction (CTI). Through extensive experiments across diverse training configurations in multimodal conditional diffusion, we consistently observe a ``winner-takes-all'' dynamic in backdoor behavior. Our results reveal that (1) attacks often collapse into subset-modality dominance, and (2) cross-modal interaction is negligible or even negative, contradicting the intuition of synergistic vulnerability. These findings highlight a critical blind spot in current assessments, suggesting that high attack success rates often mask a fundamental reliance on a subset of modalities. This establishes a principled foundation for mechanistic analysis and future defense development.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 32 of its 39 references that are in the map. First posted on arXiv 6 Mar 2026.
Builds on
32 of 39 references.
Closest ancestors · 9
Same direction, newest first
- 2025Practical, Generalizable and Robust Backdoor Attacks on Text-to-Image Diffusion Models3 Aug
- 2025Invisible Backdoor Triggers in Image Editing Model via Deep Watermarking5 Jun
- 2025Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models22 Mar
- 2025Backdoor Attacks on Discrete Graph Diffusion Models8 Mar
- 2023From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Models4 Nov
- 2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models12 Jun
- 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning7 May
- 2023Extracting Training Data from Diffusion Models30 Jan
- 2022How to Backdoor Diffusion Models?11 Dec
Other references · 23
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
- 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
- 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
- 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
- 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
- 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
- 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
11 more
- 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
- 2024OmniEdit: Building Image Editing Generalist Models Through Specialist Supervisioncited by 145
- 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
- 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
- 2023Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editingcited by 76
- 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
- 2024shapiq: Shapley Interactions for Machine Learningcited by 42
- 2022Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimizationcited by 32
- 2025A Closer Look at Multimodal Representation Collapsecited by 25
- 2024Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editingcited by 17
- 2025Improving Multimodal Learning Balance and Sufficiency through Data Remixingcited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024TrojanEdit: Multimodal Backdoor Attack Against Image Editing Model
- 2024On the Multi-modal Vulnerability of Diffusion Models
- 2026A Survey on Adversarial Attacks and Defenses for Diffusion Models Across Multiple Modalities
- 2026Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models
- 2026Scaling Exposes the Trigger: Input-Level Backdoor Detection in Text-to-Image Diffusion Models via Cross-Attention Scaling
- 2026Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models
- 2026When Backdoors Go Beyond Triggers: Semantic Drift in Diffusion Models Under Encoder Attacks
- 2026A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models