Data to 7 Oct 2026

Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models

arXiv 2404.02928 ↗First posted 2 Apr 2024cs.CR cs.AIcited by 68

On the map: Diffusion models → Robustness, privacy and adversarial aspects

also in 1 more

LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Text-to-image (T2I) models can be maliciously used to generate harmful content such as sexually explicit, unfaithful, and misleading or Not-Safe-for-Work (NSFW) images. Previous attacks largely depend on the availability of the diffusion model or involve a lengthy optimization process. In this work, we investigate a more practical and universal attack that does not require the presence of a target model and demonstrate that the high-dimensional text embedding space inherently contains NSFW concepts that can be exploited to generate harmful images. We present the Jailbreaking Prompt Attack (JPA). JPA first searches for the target malicious concepts in the text embedding space using a group of antonyms generated by ChatGPT. Subsequently, a prefix prompt is optimized in the discrete vocabulary space to align malicious concepts semantically in the text embedding space. We further introduce a soft assignment with gradient masking technique that allows us to perform gradient ascent in the discrete vocabulary space. We perform extensive experiments with open-sourced T2I models, e.g. stable-diffusion-v1-4 and closed-sourced online services, e.g. DALLE2, Midjourney with black-box safety checkers. Results show that (1) JPA bypasses both text and image safety checkers (2) while preserving high semantic alignment with the target prompt. (3) JPA demonstrates a much faster speed than previous methods and can be executed in a fully automated manner. These merits render it a valuable tool for robustness evaluation in future text-to-image generation research.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 45 of its 65 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

45 of 65 references.

Closest ancestors · 13

Same direction, newest first

1 more

2022Red-Teaming the Stable Diffusion Safety Filter3 Oct

Other references · 32

2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
2020Denoising Diffusion Probabilistic Modelscited by 20,851
2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
2022Classifier-Free Diffusion Guidancecited by 5,717
2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229

20 more

2016Improved Techniques for Training GANscited by 2,938
2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 514
2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
2021Score-based diffusion models for accelerated MRIcited by 308
2021Crystal Diffusion Variational Autoencoder for Periodic Material Generationcited by 268
2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
2022Discovering the Hidden Vocabulary of DALLE-2cited by 42
2023Federated Generative Learning with Foundation Modelscited by 23
2022TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimizationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 28; 2026: 20.

68 papers.

Sort:

By area:

2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
2024Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Conceptscited by 39
2024Perception-guided Jailbreak against Text-to-Image Modelscited by 37
2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
2025Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findingscited by 24
2024DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimizationcited by 19
2024Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearningcited by 19
2024PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximizationcited by 17

38 more

2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Modelscited by 17
2024Espresso: Robust Concept Filtering in Text-to-Image Modelscited by 14
2024Unstable Unlearning: The Hidden Risk of Concept Resurgence in Diffusion Modelscited by 14
2024SteerDiff: Steering towards Safe Text-to-Image Diffusion Modelscited by 10
2024AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion modelscited by 10
2025Jailbreaking on Text-to-Video Models via Scene Splitting Strategycited by 10
2025Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Modelscited by 9
2025Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearningcited by 9
2025Watermarking Diffusion Language Modelscited by 9
2025Red-Teaming Text-to-Image Systems by Rule-based Preference Modelingcited by 8
2024Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimizationcited by 7
2024Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Modelscited by 7
2025"I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Modelscited by 7
2025Anyone Can Jailbreak: Prompt-Based Attacks on LLMs and T2Iscited by 6
2024Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Promptingcited by 5
2026Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filterscited by 5
2025On Surjectivity of Neural Networks: Can you elicit any behavior from your model?cited by 4
2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
2025PurifyGen: A Risk-Discrimination and Semantic-Purification Model for Safe Text-to-Image Generationcited by 4
2024Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generationcited by 3
2025NSFW-Classifier Guided Prompt Sanitization for Safe Text-to-Image Generationcited by 3
2025Responsible Diffusion: A Comprehensive Survey on Safety, Ethics, and Trust in Diffusion Modelscited by 3
2026SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generationcited by 3
2024Antelope: Potent and Concealed Jailbreak Attack Strategycited by 2
2025Multimodal Prompt Decoupling Attack on the Safety Filters in Text-to-Image Modelscited by 2
2025SafeEditor: Unified MLLM for Efficient Post-hoc T2I Safety Editingcited by 2
2025Value-Aligned Prompt Moderation via Zero-Shot Agentic Rewriting for Safe Image Generationcited by 2
2025SPQR: A Multi-Dimensional Benchmark for Safety Alignment under Benign Model Adaptationcited by 2
2026Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Modelscited by 2
2026SafeRoPE: Risk-specific Head-wise Embedding Rotation for Safe Generation in Rectified Flow Transformerscited by 2
2024Mitigating Adversarial Attacks in LLMs through Defensive Suffix Generationcited by 1
2025Reconstructing Template-Memorized Images from Natural Promptscited by 1
2025ZIUM: Zero-Shot Intent-Aware Adversarial Attack on Unlearned Modelscited by 1
2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
2025Beyond Sharp Minima: Robust LLM Unlearning via Feedback-Guided Multi-Point Optimizationcited by 1
2026The Illusion of Forgetting: Attack Unlearned Diffusion via Initial Latent Variable Optimizationcited by 1
2026BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encodercited by 1
2026The Salami Slicing Threat: Exploiting Cumulative Risks in LLM Systemscited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way