mlsanityData to 14 Oct 2026

Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models

arXiv 2408.10571 ↗First posted 20 Aug 2024cs.CV cs.AIcited by 14

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

Diffusion models have revolutionized customized text-to-image generation, allowing for efficient synthesis of photos from personal data with textual descriptions. However, these advancements bring forth risks including privacy breaches and unauthorized replication of artworks. Previous researches primarily center around using prompt-specific methods to generate adversarial examples to protect personal images, yet the effectiveness of existing methods is hindered by constrained adaptability to different prompts. In this paper, we introduce a Prompt-Agnostic Adversarial Perturbation (PAP) method for customized diffusion models. PAP first models the prompt distribution using a Laplace Approximation, and then produces prompt-agnostic perturbations by maximizing a disturbance expectation based on the modeled distribution. This approach effectively tackles the prompt-agnostic attacks, leading to improved defense stability. Extensive experiments in face privacy and artistic style protection, demonstrate the superior generalization of PAP in comparison to existing techniques. Our project page is available at https://github.com/vancyland/Prompt-Agnostic-Adversarial-Perturbation-for-Customized-Diffusion-Models.github.io.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 61 of its 74 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

61 of 74 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Exploring Adversarial Attacks against Latent Diffusion Model from the Perspective of Adversarial Transferability13 Jan
  2. 2023Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model18 Dec
  3. 2023Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?30 Nov
  4. 2023Targeted Attack Improves Protection against Unauthorized Diffusion Customization7 Oct
  5. 2023Toward effective protection against diffusion based mimicry through score distillation2 Oct
  6. 2023DUAW: Data-free Universal Adversarial Watermark against Stable Diffusion Customization19 Aug
  7. 2023Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation2 Jun
  8. 2023Robust Classification via a Single Diffusion Model24 May
  9. 2023DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection23 May
  10. 2023Towards Prompt-robust Face Privacy Protection via Adversarial Decoupling Augmentation Framework6 May
  11. 2023Anti-DreamBooth: Protecting users from personalized text-to-image synthesis27 Mar
  12. 2023Raising the Cost of Malicious AI-Powered Image Editing13 Feb
2 more
  1. 2023Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples9 Feb
  2. 2022Diffusion Models for Adversarial Purification16 May

Other references · 47

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  12. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
35 more
  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  8. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  12. 2014Conditional Generative Adversarial Netscited by 2,162
  13. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  14. 2016Adversarial examples in the physical worldcited by 1,769
  15. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  16. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  17. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 355
  18. 2019HopSkipJumpAttack: A Query-Efficient Decision-Based Attackcited by 333
  19. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 244
  20. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
  21. 2023Glaze: Protecting Artists from Style Mimicry by Text-to-Image Modelscited by 205
  22. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 182
  23. 2021LowKey: Leveraging Adversarial Attacks to Protect Social Media Users from Facial Recognitioncited by 94
  24. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 89
  25. 2023ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describecited by 59
  26. 2021Boosting Adversarial Transferability through Enhanced Momentumcited by 54
  27. 2017Simple and Effective Dimensionality Reduction for Word Embeddingscited by 39
  28. 2023Security and Privacy on Generative Data in AIGC: A Surveycited by 34
  29. 2021On the Sensitivity and Stability of Model Interpretations in NLPcited by 30
  30. 2024Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learningcited by 29
  31. 2020Defending against GAN-based Deepfake Attacks via Transformation-aware Adversarial Facescited by 21
  32. 2023Challenges and Remedies to Privacy and Security in AIGC: Exploring the Potential of Privacy Computing, Blockchain, and Beyondcited by 14
  33. 2023Situating the social issues of image generation models in the model life cycle: a sociotechnical approachcited by 9
  34. 2024Projecting Points to Axes: Oriented Object Detection via Point-Axis Representationcited by 8
  35. 2023AI Imagery and the Overton Windowcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Modelscited by 7
  2. 2025DualCP: Rehearsal-Free Domain-Incremental Learning via Dual-Level Concept Prototypecited by 7
  3. 2024Grid: Omni Visual Generationcited by 5
  4. 2025GuardDoor: Safeguarding Against Malicious Diffusion Editing via Protective Backdoorscited by 5
  5. 2025Beyond Vulnerabilities: A Survey of Adversarial Attacks as Both Threats and Defenses in Computer Vision Systemscited by 4
  6. 2025Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Modelscited by 3
  7. 2026ReMoT: Reinforcement Learning with Motion Contrast Tripletscited by 3
  8. 2024Protective Perturbations against Unauthorized Data Usage in Diffusion-based Image Generationcited by 2
  9. 2025Privacy Protection Against Personalized Text-to-Image Synthesis via Cross-image Consistency Constraintscited by 2
  10. 2025EditTrack: Detecting and Attributing AI-assisted Image Editingcited by 2
  11. 2025PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoorscited by 1
  12. 2026Off-The-Shelf Image-to-Image Models Are All You Need To Defeat Image Protection Schemescited by 1
2 more
  1. 2026VOID: Defeating Unauthorized Mimicry in Latent Diffusion Modelscited by 0
  2. 2026InkShield: Writing Style Protection Against Unauthorized Handwriting Mimicrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Adversarial Protections for Diffusion Personalization: A Comprehensive Study
  2. 2025Make the Most of Everything: Further Considerations on Disrupting Diffusion-based Customization
  3. 2024Rethinking and Red-Teaming Protective Perturbation in Personalized Diffusion Models
  4. 2025CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
  5. 2023SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models
  6. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Survey
  7. 2025Towards Robust Defense against Customization via Protective Perturbation Resistant to Diffusion-based Purification
  8. 2024Visual-Friendly Concept Protection via Selective Adversarial Perturbations