mlsanityData to 7 Oct 2026

Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts

arXiv 2309.06135 ↗First posted 12 Sep 2023cs.CL cs.CVcited by 161

On the map: Diffusion models → Robustness, privacy and adversarial aspects

also in 1 more
Abstract (arXiv)

Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown remarkable ability in high-quality content generation, and become one of the representatives for the recent wave of transformative AI. Nevertheless, such advance comes with an intensifying concern about the misuse of this generative technology, especially for producing copyrighted or NSFW (i.e. not safe for work) images. Although efforts have been made to filter inappropriate images/prompts or remove undesirable concepts/styles via model fine-tuning, the reliability of these safety mechanisms against diversified problematic prompts remains largely unexplored. In this work, we propose Prompting4Debugging (P4D) as a debugging and red-teaming tool that automatically finds problematic prompts for diffusion models to test the reliability of a deployed safety mechanism. We demonstrate the efficacy of our P4D tool in uncovering new vulnerabilities of SD models with safety mechanisms. Particularly, our result shows that around half of prompts in existing safe prompting benchmarks which were originally considered "safe" can actually be manipulated to bypass many deployed safety mechanisms, including concept removal, negative prompt, and safety guidance. Our findings suggest that, without comprehensive testing, the evaluations on limited safe prompting benchmarks can lead to a false sense of safety for text-to-image models.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 34 of its 48 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

34 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Erasing Concepts from Diffusion Models13 Mar
  2. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models9 Nov
  3. 2022Red-Teaming the Stable Diffusion Safety Filter3 Oct

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
19 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2019How Can We Know What Language Models Know?cited by 953
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  10. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  11. 2019A Unified MRC Framework for Named Entity Recognitioncited by 199
  12. 2019Controllable Text-to-Image Generationcited by 187
  13. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  14. 2021Template-Based Named Entity Recognition Using BARTcited by 153
  15. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  16. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  17. 2023Red Teaming Language Model Detectors with Language Modelscited by 55
  18. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
  19. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimizationcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 46; 2025: 53; 2026: 57.

161 papers.

Sort:

By area:

  1. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 239
  2. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  3. 2024Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Modelscited by 164
  4. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  5. 2024Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Modelscited by 123
  6. 2023Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Eraserscited by 118
  7. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  8. 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 68
  9. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 68
  10. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 61
  11. 2024R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Modelcited by 56
  12. 2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 52
38 more
  1. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  2. 2024Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservationcited by 48
  3. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  4. 2024STEREO: A Two-Stage Framework for Adversarially Robust Concept Erasing from Text-to-Image Diffusion Modelscited by 39
  5. 2024RIGID: A Training-free and Model-Agnostic Framework for Robust AI-Generated Image Detectioncited by 36
  6. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  7. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  8. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  9. 2024Universal Prompt Optimizer for Safe Text-to-Image Generationcited by 27
  10. 2024Separable Multi-Concept Erasure from Diffusion Modelscited by 25
  11. 2024TraSCE: Trajectory Steering for Concept Erasurecited by 25
  12. 2024Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Modelscited by 24
  13. 2025CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Modelscited by 24
  14. 2024Fantastic Copyrighted Beasts and How (Not) to Generate Themcited by 23
  15. 2024MLLM-as-a-Judge for Image Safety without Human Labelingcited by 23
  16. 2024Pruning for Robust Concept Erasing in Diffusion Modelscited by 22
  17. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 21
  18. 2024DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimizationcited by 19
  19. 2024Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearningcited by 19
  20. 2025Training-Free Safe Denoisers for Safe Use of Diffusion Modelscited by 19
  21. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 17
  22. 2024Targeted Unlearning with Single Layer Unlearning Gradientcited by 16
  23. 2024Espresso: Robust Concept Filtering in Text-to-Image Modelscited by 14
  24. 2025TRCE: Towards Reliable Malicious Concept Erasure in Text-to-Image Diffusion Modelscited by 14
  25. 2025One Image is Worth a Thousand Words: A Usability Preservable Text-Image Collaborative Erasing Frameworkcited by 14
  26. 2025When Are Concepts Erased From Diffusion Models?cited by 14
  27. 2024Multimodal Pragmatic Jailbreak on Text-to-image Modelscited by 13
  28. 2025Minimalist Concept Erasure in Generative Modelscited by 13
  29. 2025Side Effects of Erasing Concepts from Diffusion Modelscited by 12
  30. 2025VideoEraser: Concept Erasure in Text-to-Video Diffusion Modelscited by 12
  31. 2024Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Modelscited by 11
  32. 2024Unveiling Concept Attribution in Diffusion Modelscited by 11
  33. 2025Concept Corrector: Erase concepts on the fly for text-to-image diffusion modelscited by 11
  34. 2025Revoking Amnesia: RL-based Trajectory Optimization to Resurrect Erased Concepts in Diffusion Modelscited by 11
  35. 2024SteerDiff: Steering towards Safe Text-to-Image Diffusion Modelscited by 10
  36. 2025Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppressioncited by 10
  37. 2025Jailbreaking on Text-to-Video Models via Scene Splitting Strategycited by 10
  38. 2024FLRT: Fluent Student-Teacher Redteamingcited by 9

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
  2. 2025Prompt-Based Safety Guidance Is Ineffective for Unlearned Text-to-Image Diffusion Models
  3. 2025Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimization
  4. 2023On Copyright Risks of Text-to-Image Diffusion Models
  5. 2025GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models
  6. 2024ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users
  7. 2023MMA-Diffusion: MultiModal Attack on Diffusion Models
  8. 2026Test-Time Scaling for Safe Text-Guided Image Generation via Intermediate Clean Estimates