# On the Multi-modal Vulnerability of Diffusion Models

arXiv: https://arxiv.org/abs/2402.01369 · HTML: https://mlsanity.com/paper/2402.01369/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 5 of its 36 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 5 of 5)

- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)

## Built on by (most cited first, 15 of 15)

- [Improved Techniques for Optimization-Based Jailbreaking on Large Language Models](https://mlsanity.com/paper/2405.21018/) (2024)
- [Perception-guided Jailbreak against Text-to-Image Models](https://mlsanity.com/paper/2408.10848/) (2024)
- [Adversarial Robustness for Visual Grounding of Multimodal Large Language Models](https://mlsanity.com/paper/2405.09981/) (2024)
- [FMM-Attack: A Flow-based Multi-modal Adversarial Attack on Video-based LLMs](https://mlsanity.com/paper/2403.13507/) (2024)
- [Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Models](https://mlsanity.com/paper/2504.11106/) (2025)
- [Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey](https://mlsanity.com/paper/2409.18214/) (2024)
- [AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts](https://mlsanity.com/paper/2510.24034/) (2025)
- [SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability](https://mlsanity.com/paper/2510.23960/) (2025)
- [Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generation](https://mlsanity.com/paper/2412.07249/) (2024)
- [Antelope: Potent and Concealed Jailbreak Attack Strategy](https://mlsanity.com/paper/2412.08156/) (2024)
- [JANUS: A Lightweight Framework for Jailbreaking Text-to-Image Models via Distribution Optimization](https://mlsanity.com/paper/2603.21208/) (2026)
- [Rethinking the Intermediate Features in Adversarial Attacks: Misleading Robotic Models via Adversarial Distillation](https://mlsanity.com/paper/2411.15222/) (2024)
- [Towards Dataset Copyright Evasion Attack against Personalized Text-to-Image Diffusion Models](https://mlsanity.com/paper/2505.02824/) (2025)
- [Value-Aligned Prompt Moderation via Zero-Shot Agentic Rewriting for Safe Image Generation](https://mlsanity.com/paper/2511.11693/) (2025)
- [MacPrompt: Maraconic-guided Jailbreak against Text-to-Image Models](https://mlsanity.com/paper/2601.07141/) (2026)

## Abstract (arXiv)

Diffusion models have been widely deployed in various image generation tasks, demonstrating an extraordinary connection between image and text modalities. Although prior studies have explored the vulnerability of diffusion models from the perspectives of text and image modalities separately, the current research landscape has not yet thoroughly investigated the vulnerabilities that arise from the integration of multiple modalities, specifically through the joint analysis of textual and visual features. In this paper, we are the first to visualize both text and image feature space embedded by diffusion models and observe a significant difference. The prompts are embedded chaotically in the text feature space, while in the image feature space they are clustered according to their subjects. These fascinating findings may underscore a potential misalignment in robustness between the two modalities that exists within diffusion models. Based on this observation, we propose MMP-Attack, which leverages multi-modal priors (MMP) to manipulate the generation results of diffusion models by appending a specific suffix to the original prompt. Specifically, our goal is to induce diffusion models to generate a specific object while simultaneously eliminating the original object. Our MMP-Attack shows a notable advantage over existing studies with superior manipulation capability and efficiency. Our code is publicly available at urlhttps://github.com/ydc123/MMP-Attack.