# Exploring Adversarial Attacks against Latent Diffusion Model from the Perspective of Adversarial Transferability

arXiv: https://arxiv.org/abs/2401.07087 · HTML: https://mlsanity.com/paper/2401.07087/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 26 of its 28 references that are in the map.
- First posted on arXiv 13 Jan 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 26 of 26)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples](https://mlsanity.com/paper/2302.04578/) (2023)
- [Raising the Cost of Malicious AI-Powered Image Editing](https://mlsanity.com/paper/2302.06588/) (2023)
- [Anti-DreamBooth: Protecting users from personalized text-to-image synthesis](https://mlsanity.com/paper/2303.15433/) (2023)
- [Adversarial Examples Make Strong Poisons](https://mlsanity.com/paper/2106.10807/) (2021)
- [Mist: Towards Improved Adversarial Examples for Diffusion Models](https://mlsanity.com/paper/2305.12683/) (2023)
- [On Success and Simplicity: A Second Look at Transferable Targeted Attacks](https://mlsanity.com/paper/2012.11207/) (2020)
- [Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferability](https://mlsanity.com/paper/2111.10752/) (2021)
- [Learning Transferable Adversarial Examples via Ghost Networks](https://mlsanity.com/paper/1812.03413/) (2018)
- [Toward effective protection against diffusion based mimicry through score distillation](https://mlsanity.com/paper/2311.12832/) (2023)
- [An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferability](https://mlsanity.com/paper/2308.02897/) (2023)
- [LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinity](https://mlsanity.com/paper/2207.13129/) (2022)
- [TRS: Transferability Reduced Ensemble via Encouraging Gradient Diversity and Model Smoothness](https://mlsanity.com/paper/2104.00671/) (2021)
- [SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models](https://mlsanity.com/paper/2312.07865/) (2023)
- [A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks](https://mlsanity.com/paper/2106.02105/) (2021)
- [Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability](https://mlsanity.com/paper/2307.07873/) (2023)
- [On the Robustness of Latent Diffusion Models](https://mlsanity.com/paper/2306.08257/) (2023)
- [Contrastive Language-Image Pre-training for the Italian Language](https://mlsanity.com/paper/2108.08688/) (2021)
- [JPEG Compressed Images Can Bypass Protections Against AI Editing](https://mlsanity.com/paper/2304.02234/) (2023)

## Built on by (most cited first, 5 of 5)

- [Diffusion Policy Attacker: Crafting Adversarial Attacks for Diffusion-based Policies](https://mlsanity.com/paper/2405.19424/) (2024)
- [Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models](https://mlsanity.com/paper/2408.10571/) (2024)
- [Pixel is a Barrier: Diffusion Models Are More Adversarially Robust Than We Think](https://mlsanity.com/paper/2404.13320/) (2024)
- [Protective Perturbations against Unauthorized Data Usage in Diffusion-based Image Generation](https://mlsanity.com/paper/2412.18791/) (2024)
- [VOID: Defeating Unauthorized Mimicry in Latent Diffusion Models](https://mlsanity.com/paper/2606.12263/) (2026)

## Abstract (arXiv)

Recently, many studies utilized adversarial examples (AEs) to raise the cost of malicious image editing and copyright violation powered by latent diffusion models (LDMs). Despite their successes, a few have studied the surrogate model they used to generate AEs. In this paper, from the perspective of adversarial transferability, we investigate how the surrogate model's property influences the performance of AEs for LDMs. Specifically, we view the time-step sampling in the Monte-Carlo-based (MC-based) adversarial attack as selecting surrogate models. We find that the smoothness of surrogate models at different time steps differs, and we substantially improve the performance of the MC-based AEs by selecting smoother surrogate models. In the light of the theoretical framework on adversarial transferability in image classification, we also conduct a theoretical analysis to explain why smooth surrogate models can also boost AEs for LDMs.