# Toward effective protection against diffusion based mimicry through score distillation

arXiv: https://arxiv.org/abs/2311.12832 · HTML: https://mlsanity.com/paper/2311.12832/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 29 of 29)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models](https://mlsanity.com/paper/2302.04222/) (2023)
- [Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples](https://mlsanity.com/paper/2302.04578/) (2023)
- [Raising the Cost of Malicious AI-Powered Image Editing](https://mlsanity.com/paper/2302.06588/) (2023)
- [Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness](https://mlsanity.com/paper/2302.10893/) (2023)
- [A Recipe for Watermarking Diffusion Models](https://mlsanity.com/paper/2303.10137/) (2023)
- [Diffusion Models for Imperceptible and Transferable Adversarial Attack](https://mlsanity.com/paper/2305.08192/) (2023)
- [Mist: Towards Improved Adversarial Examples for Diffusion Models](https://mlsanity.com/paper/2305.12683/) (2023)
- [Content-based Unrestricted Adversarial Attack](https://mlsanity.com/paper/2305.10665/) (2023)
- [Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllability](https://mlsanity.com/paper/2305.16494/) (2023)
- [DiffusionShield: A Watermark for Copyright Protection against Generative Diffusion Models](https://mlsanity.com/paper/2306.04642/) (2023)
- [DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection](https://mlsanity.com/paper/2305.13625/) (2023)
- [On the Robustness of Latent Diffusion Models](https://mlsanity.com/paper/2306.08257/) (2023)
- [JPEG Compressed Images Can Bypass Protections Against AI Editing](https://mlsanity.com/paper/2304.02234/) (2023)

## Built on by (most cited first, 30 of 65)

- [Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation](https://mlsanity.com/paper/2410.15618/) (2024)
- [Targeted Attack Improves Protection against Unauthorized Diffusion Customization](https://mlsanity.com/paper/2310.04687/) (2023)
- [Diffusion Policy Attacker: Crafting Adversarial Attacks for Diffusion-based Policies](https://mlsanity.com/paper/2405.19424/) (2024)
- [CDI: Copyrighted Data Identification in Diffusion Models](https://mlsanity.com/paper/2411.12858/) (2024)
- [Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models](https://mlsanity.com/paper/2408.10571/) (2024)
- [A Survey of Defenses Against AI-Generated Visual Media: Detection,Disruption, and Authentication](https://mlsanity.com/paper/2407.10575/) (2024)
- [A Gray-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapse](https://mlsanity.com/paper/2408.10901/) (2024)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [Dormant: Defending against Pose-driven Human Image Animation](https://mlsanity.com/paper/2409.14424/) (2024)
- [AdvPaint: Protecting Images from Inpainting Manipulation via Adversarial Attention Disruption](https://mlsanity.com/paper/2503.10081/) (2025)
- [Pixel is a Barrier: Diffusion Models Are More Adversarially Robust Than We Think](https://mlsanity.com/paper/2404.13320/) (2024)
- [DiffVax: Optimization-Free Image Immunization Against Diffusion-Based Editing](https://mlsanity.com/paper/2411.17957/) (2024)
- [AdLift: Lifting Adversarial Perturbations to Safeguard 3D Gaussian Splatting Assets Against Instruction-Driven Editing](https://mlsanity.com/paper/2512.07247/) (2025)
- [Imperceptible Protection against Style Imitation from Diffusion Models](https://mlsanity.com/paper/2403.19254/) (2024)
- [Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models](https://mlsanity.com/paper/2404.08030/) (2024)
- [FaceShield: Defending Facial Image against Deepfake Threats](https://mlsanity.com/paper/2412.09921/) (2024)
- [Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation](https://mlsanity.com/paper/2506.01591/) (2025)
- [Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models](https://mlsanity.com/paper/2412.11423/) (2024)
- [TarPro: Targeted Protection against Malicious Image Editing](https://mlsanity.com/paper/2503.13994/) (2025)
- [DCT-Shield: A Robust Frequency Domain Defense against Malicious Image Editing](https://mlsanity.com/paper/2504.17894/) (2025)
- [Anti-I2V: Safeguarding your photos from malicious image-to-video generation](https://mlsanity.com/paper/2603.24570/) (2026)
- [PID: Prompt-Independent Data Protection Against Latent Diffusion Models](https://mlsanity.com/paper/2406.15305/) (2024)
- [SITA: Structurally Imperceptible and Transferable Adversarial Attacks for Stylized Image Generation](https://mlsanity.com/paper/2503.19791/) (2025)
- [Exploring Adversarial Attacks against Latent Diffusion Model from the Perspective of Adversarial Transferability](https://mlsanity.com/paper/2401.07087/) (2024)
- [GuardDoor: Safeguarding Against Malicious Diffusion Editing via Protective Backdoors](https://mlsanity.com/paper/2503.03944/) (2025)
- [My Face Is Mine, Not Yours: Facial Protection Against Diffusion Model Face Swapping](https://mlsanity.com/paper/2505.15336/) (2025)
- [OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions](https://mlsanity.com/paper/2505.21724/) (2025)
- [BlurGuard: A Simple Approach for Robustifying Image Protection Against AI-Powered Editing](https://mlsanity.com/paper/2511.00143/) (2025)
- [Rethinking and Red-Teaming Protective Perturbation in Personalized Diffusion Models](https://mlsanity.com/paper/2406.18944/) (2024)
- [CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models](https://mlsanity.com/paper/2502.07225/) (2025)

## Abstract (arXiv)

While generative diffusion models excel in producing high-quality images, they can also be misused to mimic authorized images, posing a significant threat to AI systems. Efforts have been made to add calibrated perturbations to protect images from diffusion-based mimicry pipelines. However, most of the existing methods are too ineffective and even impractical to be used by individual users due to their high computation and memory requirements. In this work, we present novel findings on attacking latent diffusion models (LDM) and propose new plug-and-play strategies for more effective protection. In particular, we explore the bottleneck in attacking an LDM, discovering that the encoder module rather than the denoiser module is the vulnerable point. Based on this insight, we present our strategy using Score Distillation Sampling (SDS) to double the speed of protection and reduce memory occupation by half without compromising its strength. Additionally, we provide a robust protection strategy by counterintuitively minimizing the semantic loss, which can assist in generating more natural perturbations. Finally, we conduct extensive experiments to substantiate our findings and comprehensively evaluate our newly proposed strategies. We hope our insights and protective measures can contribute to better defense against malicious diffusion-based mimicry, advancing the development of secure AI systems. The code is available in https://github.com/xavihart/Diff-Protect