mlsanityData to 7 Oct 2026

Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning

arXiv 2305.04175 ↗First posted 7 May 2023cs.CR cs.CV cs.MMcited by 91

On the map: Diffusion models → Robustness, privacy and adversarial aspects

also in 1 more
Abstract (arXiv)

With the help of conditioning mechanisms, the state-of-the-art diffusion models have achieved tremendous success in guided image generation, particularly in text-to-image synthesis. To gain a better understanding of the training process and potential risks of text-to-image synthesis, we perform a systematic investigation of backdoor attack on text-to-image diffusion models and propose BadT2I, a general multimodal backdoor attack framework that tampers with image synthesis in diverse semantic levels. Specifically, we perform backdoor attacks on three levels of the vision semantics: Pixel-Backdoor, Object-Backdoor and Style-Backdoor. By utilizing a regularization loss, our methods efficiently inject backdoors into a large-scale text-to-image diffusion model while preserving its utility with benign inputs. We conduct empirical experiments on Stable Diffusion, the widely-used text-to-image diffusion model, demonstrating that the large-scale diffusion model can be easily backdoored within a few fine-tuning steps. We conduct additional experiments to explore the impact of different types of textual triggers, as well as the backdoor persistence during further training, providing insights for the development of backdoor defense methods. Besides, our investigation may contribute to the copyright protection of text-to-image models in the future.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 39 of its 46 references that are in the map. First posted on arXiv 7 May 2023.

Builds on

39 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023A Recipe for Watermarking Diffusion Models17 Mar
  2. 2023TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets10 Mar
  3. 2022How to Backdoor Diffusion Models?11 Dec

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
24 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  6. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  7. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  8. 2018Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooringcited by 334
  9. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  10. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 233
  11. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  12. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  13. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 211
  14. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  15. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  16. 2020Backdoor Attacks Against Deep Learning Systems in the Physical Worldcited by 129
  17. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  18. 2021Hidden Backdoors in Human-Centric Language Modelscited by 83
  19. 2021Protecting Intellectual Property of Generative Adversarial Networks from Ambiguity Attackcited by 45
  20. 2022BadDet: Backdoor Attacks on Object Detectioncited by 45
  21. 2020BAAAN: Backdoor Attacks Against Autoencoder and GAN-Based Machine Learning Modelscited by 23
  22. 2022Kallima: A Clean-label Framework for Textual Backdoor Attackscited by 22
  23. 2021The Devil is in the GAN: Backdoor Attacks and Defenses in Deep Generative Modelscited by 12
  24. 2023NCL: Textual Backdoor Defense Using Noise-augmented Contrastive Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 23; 2025: 28; 2026: 27.

91 papers.

Sort:

By area:

  1. 2023Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Modelscited by 72
  2. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  3. 2023Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Modelscited by 42
  4. 2024Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancycited by 38
  5. 2023BAGM: A Backdoor Attack for Manipulating Text-to-Image Generative Modelscited by 37
  6. 2024The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipelinecited by 33
  7. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  8. 2023Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shiftcited by 31
  9. 2024SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Modelscited by 26
  10. 2024TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoorscited by 25
  11. 2023Quantifying Bias in Text-to-Image Generative Modelscited by 16
  12. 2024SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Modelscited by 16
38 more
  1. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  2. 2023Backdooring Textual Inversion for Concept Censorshipcited by 14
  3. 2024UFID: A Unified Framework for Input-level Backdoor Detection on Diffusion Modelscited by 14
  4. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  5. 2024Backdooring Bias (B2) into Stable Diffusion Modelscited by 12
  6. 2024Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Stepcited by 11
  7. 2025Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Modelscited by 11
  8. 2024ProTIP: Probabilistic Robustness Verification on Text-to-Image Diffusion Models against Stochastic Perturbationcited by 10
  9. 2023From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Modelscited by 9
  10. 2024UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Modelscited by 9
  11. 2025BadVideo: Stealthy Backdoor Attack against Text-to-Video Generationcited by 9
  12. 2025Erased but Not Forgotten: How Backdoors Compromise Concept Erasurecited by 9
  13. 2025Efficient Input-level Backdoor Defense on Text-to-Image Synthesis via Neuron Activation Variationcited by 8
  14. 2025BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Modelcited by 7
  15. 2024Understanding Implosion in Text-to-Image Generative Modelscited by 6
  16. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
  17. 2023VA3: Virtually Assured Amplification Attack on Probabilistic Copyright Protection for Text-to-Image Generative Modelscited by 5
  18. 2024Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancementcited by 5
  19. 2025Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Modelscited by 5
  20. 2025MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schr"odinger Bridgescited by 5
  21. 2023Defending Our Privacy With Backdoorscited by 4
  22. 2025TRAP: Targeted Redirecting of Agentic Preferencescited by 4
  23. 2026When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapterscited by 4
  24. 2024The last Dance : Robust backdoor attack via diffusion models and bayesian approachcited by 3
  25. 2024Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generationcited by 3
  26. 2025A Dual-Purpose Framework for Backdoor Defense and Backdoor Amplification in Diffusion Modelscited by 3
  27. 2025Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Modelscited by 3
  28. 2025Responsible Diffusion: A Comprehensive Survey on Safety, Ethics, and Trust in Diffusion Modelscited by 3
  29. 2024Lazy Layers to Make Fine-Tuned Diffusion Models More Traceablecited by 2
  30. 2024Combinational Backdoor Attack against Customized Text-to-Image Modelscited by 2
  31. 2024On the Fairness, Diversity and Reliability of Text-to-Image Generative Modelscited by 2
  32. 2025Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Modelscited by 2
  33. 2025Parasite: A Steganography-based Backdoor Attack Framework for Diffusion Modelscited by 2
  34. 2025What Lurks Within? Concept Auditing for Shared Diffusion Models at Scalecited by 2
  35. 2025Backdoor Defense in Diffusion Models via Spatial Attention Unlearningcited by 2
  36. 2025Towards Dataset Copyright Evasion Attack against Personalized Text-to-Image Diffusion Modelscited by 2
  37. 2025BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHFcited by 2
  38. 2025Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillationcited by 2

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attacks
  2. 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis
  3. 2025PEPPER: Perception-Guided Perturbation for Robust Backdoor Defense in Text-to-Image Diffusion Models
  4. 2024T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models
  5. 2026CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models
  6. 2026BadRSSD: Backdoor Attacks on Regularized Self-Supervised Diffusion Models
  7. 2026Learning Normal Diffusion Dynamics for Backdoor Defense in Text-to-Image Models
  8. 2024Invisible Backdoor Attacks on Diffusion Models