# On the Robustness of Latent Diffusion Models

arXiv: https://arxiv.org/abs/2306.08257 · HTML: https://mlsanity.com/paper/2306.08257/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks](https://mlsanity.com/paper/1908.06281/) (2019)
- [Certified Adversarial Robustness with Additive Noise](https://mlsanity.com/paper/1809.03113/) (2018)
- [Raising the Cost of Malicious AI-Powered Image Editing](https://mlsanity.com/paper/2302.06588/) (2023)
- [Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examples](https://mlsanity.com/paper/1803.05787/) (2018)
- [Improving Adversarial Transferability via Neuron Attribution-Based Attacks](https://mlsanity.com/paper/2204.00008/) (2022)
- [A Pilot Study of Query-Free Adversarial Attack against Stable Diffusion](https://mlsanity.com/paper/2303.16378/) (2023)
- [Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization](https://mlsanity.com/paper/2303.15754/) (2023)
- [Improving the Transferability of Adversarial Samples by Path-Augmented Method](https://mlsanity.com/paper/2303.15735/) (2023)

## Built on by (most cited first, 21 of 21)

- [MMA-Diffusion: MultiModal Attack on Diffusion Models](https://mlsanity.com/paper/2311.17516/) (2023)
- [Toward effective protection against diffusion based mimicry through score distillation](https://mlsanity.com/paper/2311.12832/) (2023)
- [Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey](https://mlsanity.com/paper/2408.03400/) (2024)
- [Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing](https://mlsanity.com/paper/2411.16832/) (2024)
- [On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook](https://mlsanity.com/paper/2307.16680/) (2023)
- [A Survey of Defenses Against AI-Generated Visual Media: Detection,Disruption, and Authentication](https://mlsanity.com/paper/2407.10575/) (2024)
- [Dormant: Defending against Pose-driven Human Image Animation](https://mlsanity.com/paper/2409.14424/) (2024)
- [ProTIP: Probabilistic Robustness Verification on Text-to-Image Diffusion Models against Stochastic Perturbation](https://mlsanity.com/paper/2402.15429/) (2024)
- [Pixel is a Barrier: Diffusion Models Are More Adversarially Robust Than We Think](https://mlsanity.com/paper/2404.13320/) (2024)
- [PLA: Prompt Learning Attack against Text-to-Image Generative Models](https://mlsanity.com/paper/2508.03696/) (2025)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [FaceShield: Defending Facial Image against Deepfake Threats](https://mlsanity.com/paper/2412.09921/) (2024)
- [CtrlAttack: A Unified Attack on World-Model Control in Diffusion Models](https://mlsanity.com/paper/2603.13435/) (2026)
- [Exploring Adversarial Attacks against Latent Diffusion Model from the Perspective of Adversarial Transferability](https://mlsanity.com/paper/2401.07087/) (2024)
- [T2VAttack: Adversarial Attack on Text-to-Video Diffusion Models](https://mlsanity.com/paper/2512.23953/) (2025)
- [Backpropagation Path Search On Adversarial Transferability](https://mlsanity.com/paper/2308.07625/) (2023)
- [Protective Perturbations against Unauthorized Data Usage in Diffusion-based Image Generation](https://mlsanity.com/paper/2412.18791/) (2024)
- [Trustworthy Large Models in Vision: A Survey](https://mlsanity.com/paper/2311.09680/) (2023)
- [Cross-Cultural Fashion Design via Interactive Large Language Models and Diffusion Models](https://mlsanity.com/paper/2501.15571/) (2025)
- [Security Risk of Misalignment between Text and Image in Multi-modal Model](https://mlsanity.com/paper/2510.26105/) (2025)
- [VOID: Defeating Unauthorized Mimicry in Latent Diffusion Models](https://mlsanity.com/paper/2606.12263/) (2026)

## Abstract (arXiv)

Latent diffusion models achieve state-of-the-art performance on a variety of generative tasks, such as image synthesis and image editing. However, the robustness of latent diffusion models is not well studied. Previous works only focus on the adversarial attacks against the encoder or the output image under white-box settings, regardless of the denoising process. Therefore, in this paper, we aim to analyze the robustness of latent diffusion models more thoroughly. We first study the influence of the components inside latent diffusion models on their white-box robustness. In addition to white-box scenarios, we evaluate the black-box robustness of latent diffusion models via transfer attacks, where we consider both prompt-transfer and model-transfer settings and possible defense mechanisms. However, all these explorations need a comprehensive benchmark dataset, which is missing in the literature. Therefore, to facilitate the research of the robustness of latent diffusion models, we propose two automatic dataset construction pipelines for two kinds of image editing models and release the whole dataset. Our code and dataset are available at urlhttps://github.com/jpzhang1810/LDM-Robustness.