mlsanityData to 14 Oct 2026

Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models

arXiv 2305.10701 ↗First posted 18 May 2023cs.CVcited by 42

On the map: Diffusion models → Robustness, privacy and adversarial aspects

also in 1 more
Abstract (arXiv)

Although recent personalization methods have democratized high-resolution image synthesis by enabling swift concept acquisition with minimal examples and lightweight computation, they also present an exploitable avenue for high accessible backdoor attacks. This paper investigates a critical and unexplored aspect of text-to-image (T2I) diffusion models - their potential vulnerability to backdoor attacks via personalization. Our study focuses on a zero-day backdoor vulnerability prevalent in two families of personalization methods, epitomized by Textual Inversion and DreamBooth.Compared to traditional backdoor attacks, our proposed method can facilitate more precise, efficient, and easily accessible attacks with a lower barrier to entry. We provide a comprehensive review of personalization in T2I diffusion models, highlighting the operation and exploitation potential of this backdoor vulnerability. To be specific, by studying the prompt processing of Textual Inversion and DreamBooth, we have devised dedicated backdoor attacks according to the different ways of dealing with unseen tokens and analyzed the influence of triggers and concept images on the attack effect. Through comprehensive empirical study, we endorse the utilization of the nouveau-token backdoor attack due to its impressive effectiveness, stealthiness, and integrity, markedly outperforming the legacy-token backdoor attack.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 31 of its 52 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

31 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning7 May

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  7. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  8. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  9. 2022Diffusion Models in Vision: A Surveycited by 1,015
  10. 2020Backdoor Learning: A Surveycited by 426
  11. 2018Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooringcited by 357
  12. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 353
18 more
  1. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  2. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  3. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  4. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  5. 2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 180
  6. 2021BACKDOORL: Backdoor Attack against Competitive Reinforcement Learningcited by 68
  7. 2021Protecting Intellectual Property of Generative Adversarial Networks from Ambiguity Attackcited by 46
  8. 2021Defending against Model Stealing via Verifying Embedded External Featurescited by 46
  9. 2020FakeLocator: Robust Localization of GAN-Based Face Manipulationscited by 40
  10. 2021Dual-Key Multimodal Backdoors for Visual Question Answeringcited by 37
  11. 2023Evading DeepFake Detectors via Adversarial Statistical Consistencycited by 37
  12. 2022Multiresolution Textual Inversioncited by 34
  13. 2020FakePolisher: Making DeepFakes More Detection-Evasive by Shallow Reconstructioncited by 33
  14. 2022MOVE: Effective and Harmless Ownership Verification via Embedded External Featurescited by 21
  15. 2022Watermark Vaccine: Adversarial Attacks to Prevent Watermark Removalcited by 20
  16. 2022ALA: Naturalness-aware Adversarial Lightness Attackcited by 11
  17. 2020Dodging DeepFake Detection via Implicit Spatial-Domain Notch Filteringcited by 10
  18. 2021AdvFilter: Predictive Perturbation-aware Filtering against Adversarial Attack via Multi-domain Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 15; 2025: 15; 2026: 9.

42 papers.

Sort:

By area:

  1. 2024BadEdit: Backdooring large language models by model editingcited by 96
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  3. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 43
  4. 2024Perception-guided Jailbreak against Text-to-Image Modelscited by 40
  5. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectorycited by 36
  6. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  7. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  8. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  9. 2023Backdooring Textual Inversion for Concept Censorshipcited by 14
  10. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  11. 2024Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attackscited by 12
  12. 2025Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Modelscited by 11
30 more
  1. 2024Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Surveycited by 9
  2. 2024Texture Re-scalable Universal Adversarial Perturbationcited by 9
  3. 2025Exploring Bias in over 100 Text-to-Image Generative Modelscited by 9
  4. 2025Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Modelscited by 9
  5. 2025BadVideo: Stealthy Backdoor Attack against Text-to-Video Generationcited by 9
  6. 2025Erased but Not Forgotten: How Backdoors Compromise Concept Erasurecited by 9
  7. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  8. 2025BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Modelcited by 8
  9. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
  10. 2026When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapterscited by 4
  11. 2024The last Dance : Robust backdoor attack via diffusion models and bayesian approachcited by 3
  12. 2024Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generationcited by 3
  13. 2025Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Modelscited by 3
  14. 2026SafeRedir: Prompt Embedding Redirection for Robust Unlearning in Image Generation Modelscited by 3
  15. 2026Cert-LAS: Toward Certified Model Ownership Verification for Text-to-Image Diffusion Models via Layer-Adaptive Smoothingcited by 3
  16. 2024Combinational Backdoor Attack against Customized Text-to-Image Modelscited by 2
  17. 2024On the Fairness, Diversity and Reliability of Text-to-Image Generative Modelscited by 2
  18. 2025Anti-Aesthetics: Protecting Facial Privacy against Customized Text-to-Image Synthesiscited by 2
  19. 2025Backdoor Defense in Diffusion Models via Spatial Attention Unlearningcited by 2
  20. 2025Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Modelscited by 2
  21. 2025PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoorscited by 1
  22. 2025Practical, Generalizable and Robust Backdoor Attacks on Text-to-Image Diffusion Modelscited by 1
  23. 2026BadRSSD: Backdoor Attacks on Regularized Self-Supervised Diffusion Modelscited by 1
  24. 2026BlackMirror: Black-Box Backdoor Detection for Text-to-Image Models via Instruction-Response Deviationcited by 1
  25. 2024EnTruth: Enhancing the Traceability of Unauthorized Dataset Usage in Text-to-image Diffusion Models with Minimal and Robust Alterationscited by 0
  26. 2025CAHS-Attack: CLIP-Aware Heuristic Search Attack Method for Stable Diffusioncited by 0
  27. 2026Backdoor Sentinel: Detecting and Detoxifying Backdoors in Diffusion Models via Temporal Noise Consistencycited by 0
  28. 2026Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Modelscited by 0
  29. 2026LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Modelscited by 0
  30. 2026BadAction: Backdoor Attacks on Interactive Video Generation via Action-Guided Triggerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis
  2. 2026Semantic-level Backdoor Attack against Text-to-Image Diffusion Models
  3. 2024T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models
  4. 2025PEPPER: Perception-Guided Perturbation for Robust Backdoor Defense in Text-to-Image Diffusion Models
  5. 2025Efficient Input-level Backdoor Defense on Text-to-Image Synthesis via Neuron Activation Variation
  6. 2026Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models
  7. 2024Backdooring Bias (B2) into Stable Diffusion Models
  8. 2023Anti-DreamBooth: Protecting users from personalized text-to-image synthesis