# Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models

arXiv: https://arxiv.org/abs/2404.15081 · HTML: https://mlsanity.com/paper/2404.15081/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/1909.11764/) (2019)
- [SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization](https://mlsanity.com/paper/1911.03437/) (2019)
- [Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models](https://mlsanity.com/paper/2302.04222/) (2023)
- [You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle](https://mlsanity.com/paper/1905.00877/) (2019)
- [Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples](https://mlsanity.com/paper/2302.04578/) (2023)
- [Raising the Cost of Malicious AI-Powered Image Editing](https://mlsanity.com/paper/2302.06588/) (2023)
- [Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation](https://mlsanity.com/paper/2306.01902/) (2023)
- [Generative Watermarking Against Unauthorized Subject-Driven Image Synthesis](https://mlsanity.com/paper/2306.07754/) (2023)
- [FT-Shield: A Watermark Against Unauthorized Fine-tuning in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2310.02401/) (2023)

## Built on by (most cited first, 18 of 18)

- [A Survey of Defenses Against AI-Generated Visual Media: Detection,Disruption, and Authentication](https://mlsanity.com/paper/2407.10575/) (2024)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [AdvPaint: Protecting Images from Inpainting Manipulation via Adversarial Attention Disruption](https://mlsanity.com/paper/2503.10081/) (2025)
- [Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models](https://mlsanity.com/paper/2412.11423/) (2024)
- [DeContext as Defense: Safe Image Editing in Diffusion Transformers](https://mlsanity.com/paper/2512.16625/) (2025)
- [Understanding Implosion in Text-to-Image Generative Models](https://mlsanity.com/paper/2409.12314/) (2024)
- [A Comprehensive Survey on Visual Concept Mining in Text-to-image Diffusion Models](https://mlsanity.com/paper/2503.13576/) (2025)
- [BadWorld: Adversarial Attacks on World Models](https://mlsanity.com/paper/2606.16519/) (2026)
- [Towards Robust Defense against Customization via Protective Perturbation Resistant to Diffusion-based Purification](https://mlsanity.com/paper/2509.13922/) (2025)
- [Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models](https://mlsanity.com/paper/2511.01307/) (2025)
- [Anti-Aesthetics: Protecting Facial Privacy against Customized Text-to-Image Synthesis](https://mlsanity.com/paper/2504.12129/) (2025)
- [An h-space Based Adversarial Attack for Protection Against Few-shot Personalization](https://mlsanity.com/paper/2507.17554/) (2025)
- [Structure Disruption: Subverting Malicious Diffusion-Based Inpainting via Self-Attention Query Perturbation](https://mlsanity.com/paper/2505.19425/) (2025)
- [Anti-Tamper Protection for Unauthorized Individual Image Generation](https://mlsanity.com/paper/2508.06325/) (2025)
- [Three-in-One: Robust Enhanced Universal Transferable Anti-Facial Retrieval in Online Social Networks](https://mlsanity.com/paper/2412.09692/) (2024)
- [AWPD: Frequency Shield Network for Agnostic Watermark Presence Detection](https://mlsanity.com/paper/2603.06723/) (2026)
- [Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2605.19698/) (2026)
- [VOID: Defeating Unauthorized Mimicry in Latent Diffusion Models](https://mlsanity.com/paper/2606.12263/) (2026)

## Abstract (arXiv)

Diffusion models (DMs) embark a new era of generative modeling and offer more opportunities for efficient generating high-quality and realistic data samples. However, their widespread use has also brought forth new challenges in model security, which motivates the creation of more effective adversarial attackers on DMs to understand its vulnerability. We propose CAAT, a simple but generic and efficient approach that does not require costly training to effectively fool latent diffusion models (LDMs). The approach is based on the observation that cross-attention layers exhibits higher sensitivity to gradient change, allowing for leveraging subtle perturbations on published images to significantly corrupt the generated images. We show that a subtle perturbation on an image can significantly impact the cross-attention layers, thus changing the mapping between text and image during the fine-tuning of customized diffusion models. Extensive experiments demonstrate that CAAT is compatible with diverse diffusion models and outperforms baseline attack methods in a more effective (more noise) and efficient (twice as fast as Anti-DreamBooth and Mist) manner.