mlsanityData to 14 Oct 2026

VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models

arXiv 2306.06874 ↗First posted 12 Jun 2023cs.CR cs.CV cs.LGcited by 68

On the map: Diffusion models → Robustness, privacy and adversarial aspects

also in 1 more
Abstract (arXiv)

Diffusion Models (DMs) are state-of-the-art generative models that learn a reversible corruption process from iterative noise addition and denoising. They are the backbone of many generative AI applications, such as text-to-image conditional generation. However, recent studies have shown that basic unconditional DMs (e.g., DDPM and DDIM) are vulnerable to backdoor injection, a type of output manipulation attack triggered by a maliciously embedded pattern at model input. This paper presents a unified backdoor attack framework (VillanDiffusion) to expand the current scope of backdoor analysis for DMs. Our framework covers mainstream unconditional and conditional DMs (denoising-based and score-based) and various training-free samplers for holistic evaluations. Experiments show that our unified framework facilitates the backdoor analysis of different DM configurations and provides new insights into caption-based backdoor attacks on DMs. Our code is available on GitHub: urlhttps://github.com/IBM/villandiffusion

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 51 of its 57 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

51 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets10 Mar
  2. 2022How to Backdoor Diffusion Models?11 Dec

Other references · 49

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
37 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  7. 2022Video Diffusion Modelscited by 2,305
  8. 2015Variational Inference with Normalizing Flowscited by 2,170
  9. 2016Density estimation using Real NVPcited by 2,042
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  12. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  13. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  14. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  15. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  16. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  17. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  18. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  19. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  20. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  21. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  22. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  23. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  24. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 450
  25. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  26. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  27. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 334
  28. 2023Imitating Human Behaviour with Diffusion Modelscited by 263
  29. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  30. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speechcited by 157
  31. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  32. 2019Fair Generative Modeling via Weak Supervisioncited by 114
  33. 2021Talk-to-Edit: Fine-Grained Facial Editing via Dialogcited by 102
  34. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  35. 2022VIDM: Video Implicit Diffusion Modelscited by 78
  36. 2019Bias Correction of Learned Generative Models using Likelihood-Free Importance Weightingcited by 70
  37. 2020AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flowscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 20; 2025: 23; 2026: 20.

68 papers.

Sort:

By area:

  1. 2023Black-box Membership Inference Attacks against Fine-tuned Diffusion Modelscited by 35
  2. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  3. 2024T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Modelscited by 29
  4. 2024TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoorscited by 25
  5. 2025REFINE: Inversion-Free Backdoor Defense via Model Reprogrammingcited by 22
  6. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  7. 2024Diff-Cleanse: Identifying and Mitigating Backdoor Attacks in Diffusion Modelscited by 13
  8. 2024Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attackscited by 12
  9. 2024Robust and Transferable Backdoor Attacks Against Deep Image Compression With Selective Frequency Priorcited by 11
  10. 2023From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Modelscited by 9
  11. 2024Invisible Backdoor Attacks on Diffusion Modelscited by 9
  12. 2024UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Modelscited by 9
38 more
  1. 2025BadVideo: Stealthy Backdoor Attack against Text-to-Video Generationcited by 9
  2. 2025Erased but Not Forgotten: How Backdoors Compromise Concept Erasurecited by 9
  3. 2023Model Selection of Anomaly Detectors in the Absence of Labeled Validation Datacited by 8
  4. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  5. 2025BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Modelcited by 8
  6. 2025Efficient Input-level Backdoor Defense on Text-to-Image Synthesis via Neuron Activation Variationcited by 8
  7. 2024Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Modelscited by 7
  8. 2024PureDiffusion: Using Backdoor to Counter Backdoor in Generative Diffusion Modelscited by 6
  9. 2025MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schr"odinger Bridgescited by 6
  10. 2025What Lurks Within? Concept Auditing for Shared Diffusion Models at Scalecited by 4
  11. 2026When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapterscited by 4
  12. 2024The last Dance : Robust backdoor attack via diffusion models and bayesian approachcited by 3
  13. 2024Trading Devil: Robust backdoor attack via Stochastic investment models and Bayesian approachcited by 3
  14. 2024Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generationcited by 3
  15. 2025A Dual-Purpose Framework for Backdoor Defense and Backdoor Amplification in Diffusion Modelscited by 3
  16. 2025Responsible Diffusion: A Comprehensive Survey on Safety, Ethics, and Trust in Diffusion Modelscited by 3
  17. 2024Detecting Dataset Abuse in Fine-Tuning Stable Diffusion Models for Text-to-Image Synthesiscited by 2
  18. 2025Hiding Images in Diffusion Models by Editing Learned Score Functionscited by 2
  19. 2025Parasite: A Steganography-based Backdoor Attack Framework for Diffusion Modelscited by 2
  20. 2025Backdoor Defense in Diffusion Models via Spatial Attention Unlearningcited by 2
  21. 2025Towards Dataset Copyright Evasion Attack against Personalized Text-to-Image Diffusion Modelscited by 2
  22. 2025BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHFcited by 2
  23. 2025Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillationcited by 2
  24. 2026Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Modelscited by 2
  25. 2024EmoAttack: Emotion-to-Image Diffusion Models for Emotional Backdoor Generationcited by 1
  26. 2024CAT: Concept-level backdoor ATtacks for Concept Bottleneck Modelscited by 1
  27. 2024How to Backdoor Consistency Models?cited by 1
  28. 2024TrojFlow: Flow Models are Natural Targets for Trojan Attackscited by 1
  29. 2025Class-Conditional Neural Polarizer: A Lightweight and Effective Backdoor Defense by Purifying Poisoned Featurescited by 1
  30. 2025Gungnir: Exploiting Stylistic Features in Images for Backdoor Attacks on Diffusion Modelscited by 1
  31. 2025Invisible Backdoor Triggers in Image Editing Model via Deep Watermarkingcited by 1
  32. 2025Practical, Generalizable and Robust Backdoor Attacks on Text-to-Image Diffusion Modelscited by 1
  33. 2025Data-Chain Backdoor: Do You Trust Diffusion Models as Generative Data Supplier?cited by 1
  34. 2026BadRSSD: Backdoor Attacks on Regularized Self-Supervised Diffusion Modelscited by 1
  35. 2026Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Modelscited by 1
  36. 2023Trustworthy Large Models in Vision: A Surveycited by 0
  37. 2024Manipulating and Mitigating Generative Model Biases without Retrainingcited by 0
  38. 2025BadBlocks: Low-Cost and Stealthy Backdoor Attacks Tailored for Text-to-Image Diffusion Modelscited by 0

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift
  2. 2024Watch the Watcher! Backdoor Attacks on Security-Enhancing Diffusion Models
  3. 2024UFID: A Unified Framework for Input-level Backdoor Detection on Diffusion Models
  4. 2024DisDet: Exploring Detectability of Backdoor Attack on Diffusion Models
  5. 2026TEMPO-Diffusion: Temporally Exposed Malicious Poisoning of Diffusion Models
  6. 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning
  7. 2023Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data
  8. 2025Backdoor Attacks on Discrete Graph Diffusion Models