mlsanityData to 14 Oct 2026

Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models

arXiv 2305.13873 ↗First posted 23 May 2023cs.CV cs.CR cs.CYcited by 175

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

State-of-the-art Text-to-Image models like Stable Diffusion and DALLE·2 are revolutionizing how people generate visual content. At the same time, society has serious concerns about how adversaries can exploit such models to generate unsafe images. In this work, we focus on demystifying the generation of unsafe images and hateful memes from Text-to-Image models. We first construct a typology of unsafe images consisting of five categories (sexually explicit, violent, disturbing, hateful, and political). Then, we assess the proportion of unsafe images generated by four advanced Text-to-Image models using four prompt datasets. We find that these models can generate a substantial percentage of unsafe images; across four models and four prompt datasets, 14.56% of all generated images are unsafe. When comparing the four models, we find different risk levels, with Stable Diffusion being the most prone to generating unsafe content (18.92% of all generated images are unsafe). Given Stable Diffusion's tendency to generate more unsafe content, we evaluate its potential to generate hateful meme variants if exploited by an adversary to attack a specific individual or community. We employ three image editing methods, DreamBooth, Textual Inversion, and SDEdit, which are supported by Stable Diffusion. Our evaluation result shows that 24% of the generated images using DreamBooth are hateful meme variants that present the features of the original hateful meme and the target individual/community; these generated images are comparable to hateful meme variants collected from the real world. Overall, our results demonstrate that the danger of large-scale generation of unsafe images is imminent. We discuss several mitigating measures, such as curating training data, regulating prompts, and implementing safety filters, and encourage better safeguard tools to be developed to prevent unsafe generation.

Key facts. Cited by 175 papers in the map (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 35 of its 66 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

35 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models9 Nov
  2. 2022Red-Teaming the Stable Diffusion Safety Filter3 Oct

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
21 more
  1. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  2. 2015YFCC100M: The New Data in Multimedia Researchcited by 235
  3. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
  4. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  5. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 155
  6. 2022A Taxonomy of Prompt Modifiers for Text-To-Image Generationcited by 55
  7. 2020Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memescited by 51
  8. 2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 51
  9. 2018On the Origins of Memes by Means of Fringe Web Communitiescited by 47
  10. 2020Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solutioncited by 41
  11. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  12. 2022Best Prompts for Text-to-Image Models and How to Find Themcited by 35
  13. 2019Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderationcited by 29
  14. 2018A Quantitative Approach to Understanding Online Antisemitismcited by 26
  15. 2022On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learningcited by 26
  16. 2020Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Boardcited by 24
  17. 2016Kek, Cucks, and God Emperor Trump: A Measurement Study of 4chan's Politically Incorrect Forum and Its Effects on the Webcited by 15
  18. 2020Do Platform Migrations Compromise Content Moderation? Evidence from r/The_Donald and r/Incelscited by 15
  19. 2022Understanding and Detecting Hateful Content using Contrastive Learningcited by 13
  20. 2022On Xing Tian and the Perseverance of Anti-China Sentiment Onlinecited by 10
  21. 2018"You Know What to Do": Proactive Detection of YouTube Videos Targeted by Coordinated Hate Attackscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 57; 2025: 57; 2026: 48.

175 papers.

Sort:

By area:

  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  2. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 250
  3. 2023MMA-Diffusion: MultiModal Attack on Diffusion Modelscited by 194
  4. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  5. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  6. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 95
  7. 2023Circumventing Concept Erasure Methods For Text-to-Image Generative Modelscited by 85
  8. 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 72
  9. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 71
  10. 2024SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Modelscited by 66
  11. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  12. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 62
38 more
  1. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  2. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  3. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 53
  4. 2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 53
  5. 2024Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservationcited by 50
  6. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 48
  7. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 47
  8. 2024Voice Jailbreak Attacks Against GPT-4ocited by 42
  9. 2024Perception-guided Jailbreak against Text-to-Image Modelscited by 40
  10. 2024ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Userscited by 35
  11. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 35
  12. 2023SurrogatePrompt: Bypassing the Safety Filter of Text-to-Image Models via Substitutioncited by 34
  13. 2024Malla: Demystifying Real-world Large Language Model Integrated Malicious Servicescited by 33
  14. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  15. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 32
  16. 2024Universal Prompt Optimizer for Safe Text-to-Image Generationcited by 28
  17. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  18. 2024PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMscited by 21
  19. 2024Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generationcited by 20
  20. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 20
  21. 2023IMMA: Immunizing text-to-image Models against Malicious Adaptationcited by 18
  22. 2025PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Modelscited by 17
  23. 2024Harm Amplification in Text-to-Image Modelscited by 15
  24. 2024Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policiescited by 15
  25. 2024Unstable Unlearning: The Hidden Risk of Concept Resurgence in Diffusion Modelscited by 15
  26. 2025Reason2Attack: Jailbreaking Text-to-Image Models via LLM Reasoningcited by 15
  27. 2024Groot: Adversarial Testing for Generative Text-to-Image Models with Tree-based Semantic Transformationcited by 14
  28. 2024Espresso: Robust Concept Filtering in Text-to-Image Modelscited by 14
  29. 2025JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoringcited by 14
  30. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  31. 2024Multimodal Pragmatic Jailbreak on Text-to-image Modelscited by 13
  32. 2025Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimizationcited by 13
  33. 2025Minimalist Concept Erasure in Generative Modelscited by 13
  34. 2023Harnessing LLM to Attack LLM-Guarded Text-to-Image Modelscited by 12
  35. 2024Replication in Visual Diffusion Models: A Survey and Outlookcited by 11
  36. 2025Concept Corrector: Erase concepts on the fly for text-to-image diffusion modelscited by 11
  37. 2026When Safety Collides: Resolving Multi-Category Harmful Conflicts in Text-to-Image Diffusion via Adaptive Safety Guidancecited by 11
  38. 2024Towards Understanding Unsafe Video Generationcited by 10

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024When Image Generation Goes Wrong: A Safety Analysis of Stable Diffusion Models
  2. 2024Exploring the Boundaries of Content Moderation in Text-to-Image Generation
  3. 2024Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Models
  4. 2025Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study
  5. 2025Prompt-Based Safety Guidance Is Ineffective for Unlearned Text-to-Image Diffusion Models
  6. 2025Training-Free Safe Text Embedding Guidance for Text-to-Image Diffusion Models
  7. 2025Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions
  8. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts