# The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipeline

arXiv: https://arxiv.org/abs/2401.04136 · HTML: https://mlsanity.com/paper/2401.04136/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 49 of its 71 references that are in the map.
- First posted on arXiv 7 Jan 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 30 of 49)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.06503/) (2022)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Poisoning Web-Scale Training Datasets is Practical](https://mlsanity.com/paper/2302.10149/) (2023)
- [Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise](https://mlsanity.com/paper/2208.09392/) (2022)
- [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)

## Built on by (most cited first, 30 of 33)

- [Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey](https://mlsanity.com/paper/2408.03400/) (2024)
- [UFID: A Unified Framework for Input-level Backdoor Detection on Diffusion Models](https://mlsanity.com/paper/2404.01101/) (2024)
- [Evaluating and Mitigating IP Infringement in Visual Generative AI](https://mlsanity.com/paper/2406.04662/) (2024)
- [Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Models](https://mlsanity.com/paper/2404.13706/) (2024)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.09669/) (2025)
- [ProTIP: Probabilistic Robustness Verification on Text-to-Image Diffusion Models against Stochastic Perturbation](https://mlsanity.com/paper/2402.15429/) (2024)
- [BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Model](https://mlsanity.com/paper/2502.11798/) (2025)
- [Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling](https://mlsanity.com/paper/2505.21074/) (2025)
- [A Systematic Review of Poisoning Attacks Against Large Language Models](https://mlsanity.com/paper/2506.06518/) (2025)
- [CopyrightShield: Enhancing Diffusion Model Security against Copyright Infringement Attacks](https://mlsanity.com/paper/2412.01528/) (2024)
- [MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schr"odinger Bridges](https://mlsanity.com/paper/2505.08809/) (2025)
- [Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning](https://mlsanity.com/paper/2506.14913/) (2025)
- [Improving Diffusion Models's Data-Corruption Resistance using Scheduled Pseudo-Huber Loss](https://mlsanity.com/paper/2403.16728/) (2024)
- [Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP](https://mlsanity.com/paper/2412.00727/) (2024)
- [CopyJudge: Automated Copyright Infringement Identification and Mitigation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2502.15278/) (2025)
- [The last Dance : Robust backdoor attack via diffusion models and bayesian approach](https://mlsanity.com/paper/2402.05967/) (2024)
- [Trading Devil: Robust backdoor attack via Stochastic investment models and Bayesian approach](https://mlsanity.com/paper/2406.10719/) (2024)
- [A Dual-Purpose Framework for Backdoor Defense and Backdoor Amplification in Diffusion Models](https://mlsanity.com/paper/2502.19047/) (2025)
- [PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Models](https://mlsanity.com/paper/2503.07697/) (2025)
- [A Linear Approach to Data Poisoning](https://mlsanity.com/paper/2505.15175/) (2025)
- [ME: Trigger Element Combination Backdoor Attack on Copyright Infringement](https://mlsanity.com/paper/2506.10776/) (2025)
- [TrojanEdit: Multimodal Backdoor Attack Against Image Editing Model](https://mlsanity.com/paper/2411.14681/) (2024)
- [Safety-Efficacy Trade Off: Robustness against Data-Poisoning](https://mlsanity.com/paper/2602.00822/) (2026)
- [TWIG: Two-Step Image Generation using Segmentation Masks in Diffusion Models](https://mlsanity.com/paper/2504.14933/) (2025)
- [Attacks on Approximate Caches in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.20424/) (2025)
- [A Unified Perspective on Adversarial Membership Manipulation in Vision Models](https://mlsanity.com/paper/2604.02780/) (2026)
- [Yes, But Not Always. Generative AI Needs Nuanced Opt-in](https://mlsanity.com/paper/2604.09413/) (2026)
- [Beyond the False Trade-off: Adaptive EWC for Stealthy and Generalizable T2I Backdoors](https://mlsanity.com/paper/2605.08280/) (2026)
- [Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2605.19698/) (2026)

## Abstract (arXiv)

The commercialization of text-to-image diffusion models (DMs) brings forth potential copyright concerns. Despite numerous attempts to protect DMs from copyright issues, the vulnerabilities of these solutions are underexplored. In this study, we formalized the Copyright Infringement Attack on generative AI models and proposed a backdoor attack method, SilentBadDiffusion, to induce copyright infringement without requiring access to or control over training processes. Our method strategically embeds connections between pieces of copyrighted information and text references in poisoning data while carefully dispersing that information, making the poisoning data inconspicuous when integrated into a clean dataset. Our experiments show the stealth and efficacy of the poisoning data. When given specific text prompts, DMs trained with a poisoning ratio of 0.20% can produce copyrighted images. Additionally, the results reveal that the more sophisticated the DMs are, the easier the success of the attack becomes. These findings underline potential pitfalls in the prevailing copyright protection strategies and underscore the necessity for increased scrutiny to prevent the misuse of DMs.