# Disrupting Diffusion: Token-Level Attention Erasure Attack against Diffusion-based Customization

arXiv: https://arxiv.org/abs/2405.20584 · HTML: https://mlsanity.com/paper/2405.20584/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 8 of its 9 references that are in the map.
- First posted on arXiv 31 May 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 8 of 8)

- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SimSwap: An Efficient Framework For High Fidelity Face Swapping](https://mlsanity.com/paper/2106.06340/) (2021)
- [CMUA-Watermark: A Cross-Model Universal Adversarial Watermark for Combating Deepfakes](https://mlsanity.com/paper/2105.10872/) (2021)
- [FakeTagger: Robust Safeguards against DeepFake Dissemination via Provenance Tracking](https://mlsanity.com/paper/2009.09869/) (2020)
- [SepMark: Deep Separable Watermarking for Unified Source Tracing and Deepfake Detection](https://mlsanity.com/paper/2305.06321/) (2023)
- [OPOM: Customized Invisible Cloak towards Face Privacy Protection](https://mlsanity.com/paper/2205.11981/) (2022)
- [Deepfake Video Detection with Spatiotemporal Dropout Transformer](https://mlsanity.com/paper/2207.06612/) (2022)
- [Deep Convolutional Pooling Transformer for Deepfake Detection](https://mlsanity.com/paper/2209.05299/) (2022)

## Built on by (most cited first, 15 of 15)

- [DeContext as Defense: Safe Image Editing in Diffusion Transformers](https://mlsanity.com/paper/2512.16625/) (2025)
- [AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts](https://mlsanity.com/paper/2510.24034/) (2025)
- [BadWorld: Adversarial Attacks on World Models](https://mlsanity.com/paper/2606.16519/) (2026)
- [Evaluating Adversarial Protections for Diffusion Personalization: A Comprehensive Study](https://mlsanity.com/paper/2507.03953/) (2025)
- [Privacy Protection in Personalized Diffusion Models via Targeted Cross-Attention Adversarial Attack](https://mlsanity.com/paper/2411.16437/) (2024)
- [Privacy Protection Against Personalized Text-to-Image Synthesis via Cross-image Consistency Constraints](https://mlsanity.com/paper/2504.12747/) (2025)
- [An h-space Based Adversarial Attack for Protection Against Few-shot Personalization](https://mlsanity.com/paper/2507.17554/) (2025)
- [Bypassing Copyright Protection in Diffusion-based Customization via Two-Stage Latent Feature Optimization](https://mlsanity.com/paper/2606.09909/) (2026)
- [Adversarial Attacks for Good: A Survey of Proactive Protection across the Visual Content Lifecycle](https://mlsanity.com/paper/2608.04314/) (2026)
- [PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors](https://mlsanity.com/paper/2502.16167/) (2025)
- [Targeted Data Protection for Diffusion Model by Matching Training Trajectory](https://mlsanity.com/paper/2512.10433/) (2025)
- [VOID: Defeating Unauthorized Mimicry in Latent Diffusion Models](https://mlsanity.com/paper/2606.12263/) (2026)
- [ChronoLock: Protecting Videos from Unauthorized Text-to-Video Personalization](https://mlsanity.com/paper/2606.21146/) (2026)
- [Generating Multi-view Adversarial Examples for Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2608.20748/) (2026)
- [A Survey on Adversarial Attacks and Defenses for Diffusion Models Across Multiple Modalities](https://mlsanity.com/paper/2609.05503/) (2026)

## Abstract (arXiv)

With the development of diffusion-based customization methods like DreamBooth, individuals now have access to train the models that can generate their personalized images. Despite the convenience, malicious users have misused these techniques to create fake images, thereby triggering a privacy security crisis. In light of this, proactive adversarial attacks are proposed to protect users against customization. The adversarial examples are trained to distort the customization model's outputs and thus block the misuse. In this paper, we propose DisDiff (Disrupting Diffusion), a novel adversarial attack method to disrupt the diffusion model outputs. We first delve into the intrinsic image-text relationships, well-known as cross-attention, and empirically find that the subject-identifier token plays an important role in guiding image generation. Thus, we propose the Cross-Attention Erasure module to explicitly "erase" the indicated attention maps and disrupt the text guidance. Besides,we analyze the influence of the sampling process of the diffusion model on Projected Gradient Descent (PGD) attack and introduce a novel Merit Sampling Scheduler to adaptively modulate the perturbation updating amplitude in a step-aware manner. Our DisDiff outperforms the state-of-the-art methods by 12.75% of FDFR scores and 7.25% of ISM scores across two facial benchmarks and two commonly used prompts on average.