mlsanityData to 7 Oct 2026

SneakyPrompt: Jailbreaking Text-to-image Generative Models

arXiv 2305.12082 ↗First posted 20 May 2023cs.LGcited by 175

On the map: Diffusion models → Robustness, privacy and adversarial aspects

also in 1 more
Abstract (arXiv)

Text-to-image generative models such as Stable Diffusion and DALL·E raise many ethical concerns due to the generation of harmful images such as Not-Safe-for-Work (NSFW) ones. To address these ethical concerns, safety filters are often adopted to prevent the generation of NSFW images. In this work, we propose SneakyPrompt, the first automated attack framework, to jailbreak text-to-image generative models such that they generate NSFW images even if safety filters are adopted. Given a prompt that is blocked by a safety filter, SneakyPrompt repeatedly queries the text-to-image generative model and strategically perturbs tokens in the prompt based on the query results to bypass the safety filter. Specifically, SneakyPrompt utilizes reinforcement learning to guide the perturbation of tokens. Our evaluation shows that SneakyPrompt successfully jailbreaks DALL·E 2 with closed-box safety filters to generate NSFW images. Moreover, we also deploy several state-of-the-art, open-source safety filters on a Stable Diffusion model. Our evaluation shows that SneakyPrompt not only successfully generates NSFW images, but also outperforms existing text adversarial attacks when extended to jailbreak text-to-image generative models, in terms of both the number of queries and qualities of the generated NSFW images. SneakyPrompt is open-source and available at this repository: urlhttps://github.com/Yuchen413/text2imagesafety.

Key facts. Cited by 175 papers in the map (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 29 of its 49 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

29 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Are Diffusion Models Vulnerable to Membership Inference Attacks?2 Feb
  2. 2023Extracting Training Data from Diffusion Models30 Jan
  3. 2022Red-Teaming the Stable Diffusion Safety Filter3 Oct

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
14 more
  1. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  2. 2018Generating Natural Language Adversarial Examplescited by 422
  3. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  4. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  5. 2015Generating Images from Captions with Attentioncited by 135
  6. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  7. 2021Practical Blind Membership Inference Attack via Differential Comparisonscited by 73
  8. 2020PatchAttack: A Black-box Texture-based Attack with Reinforcement Learningcited by 67
  9. 2022Membership Inference Attacks Against Text-to-image Generation Modelscited by 66
  10. 2020Actor-Critic Reinforcement Learning for Control with Stability Guaranteecited by 39
  11. 2020Text-to-Image Generation Grounded by Fine-Grained User Attentioncited by 36
  12. 2022Adversarial Attacks on Image Generation With Made-Up Wordscited by 29
  13. 2022Character-level White-Box Adversarial Attacks against Transformers via Attachable Subwords Substitutioncited by 28
  14. 2019Identifying Model Weakness with Adversarial Examinercited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 44; 2025: 66; 2026: 63.

175 papers.

Sort:

By area:

  1. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  2. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  3. 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 68
  4. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 68
  5. 2024SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Modelscited by 64
  6. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 61
  7. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Benchcited by 61
  8. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  9. 2024Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservationcited by 48
  10. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  11. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  12. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 39
38 more
  1. 2024Perception-guided Jailbreak against Text-to-Image Modelscited by 37
  2. 2025Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Themcited by 34
  3. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  4. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  5. 2024Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreakingcited by 30
  6. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  7. 2024Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Modelscited by 24
  8. 2024Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generationcited by 20
  9. 2024DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimizationcited by 19
  10. 2024Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearningcited by 19
  11. 2025T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attackscited by 18
  12. 2024PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximizationcited by 17
  13. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 17
  14. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 17
  15. 2025PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Modelscited by 16
  16. 2024Espresso: Robust Concept Filtering in Text-to-Image Modelscited by 14
  17. 2024Unstable Unlearning: The Hidden Risk of Concept Resurgence in Diffusion Modelscited by 14
  18. 2025Minimalist Concept Erasure in Generative Modelscited by 13
  19. 2024Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Stepcited by 11
  20. 2024Unveiling Concept Attribution in Diffusion Modelscited by 11
  21. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  22. 2023Harnessing LLM to Attack LLM-Guarded Text-to-Image Modelscited by 10
  23. 2024SteerDiff: Steering towards Safe Text-to-Image Diffusion Modelscited by 10
  24. 2024AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion modelscited by 10
  25. 2024The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Modelscited by 9
  26. 2024DiffGuard: Text-Based Safety Checker for Diffusion Modelscited by 9
  27. 2025Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Modelscited by 9
  28. 2024Hiding and Recovering Knowledge in Text-to-Image Diffusion Models via Learnable Promptscited by 8
  29. 2025Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Modelscited by 8
  30. 2025BadMoE: Backdooring Mixture-of-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Expertscited by 8
  31. 2025Red-Teaming Text-to-Image Systems by Rule-based Preference Modelingcited by 8
  32. 2025Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuningcited by 8
  33. 2025PLA: Prompt Learning Attack against Text-to-Image Generative Modelscited by 8
  34. 2024Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimizationcited by 7
  35. 2024Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Modelscited by 7
  36. 2024AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Modelscited by 7
  37. 2025Mitigating Sexual Content Generation via Embedding Distortion in Text-conditioned Diffusion Modelscited by 7
  38. 2025SafeText: Safe Text-to-image Models via Aligning the Text Encodercited by 7

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Pragmatic Jailbreak on Text-to-image Models
  2. 2026TYPO: Instruction-Dense Visual Jailbreaks against Commercial Closed-Source Image-Generation Models
  3. 2025SecureT2I: No More Unauthorized Manipulation on AI Generated Images from Prompts
  4. 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis
  5. 2026Test-Time Scaling for Safe Text-Guided Image Generation via Intermediate Clean Estimates
  6. 2024ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users
  7. 2025Reason2Attack: Jailbreaking Text-to-Image Models via LLM Reasoning
  8. 2023Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models