# SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models

arXiv: https://arxiv.org/abs/2404.06666 · HTML: https://mlsanity.com/paper/2404.06666/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?](https://mlsanity.com/paper/2310.10012/) (2023)
- [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://mlsanity.com/paper/2305.12082/) (2023)
- [Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models](https://mlsanity.com/paper/2305.13873/) (2023)
- [Circumventing Concept Erasure Methods For Text-to-Image Generative Models](https://mlsanity.com/paper/2308.01508/) (2023)
- [U-Net Transformer: Self and Cross Attention for Medical Image Segmentation](https://mlsanity.com/paper/2103.06104/) (2021)
- [Normalized and Geometry-Aware Self-Attention Network for Image Captioning](https://mlsanity.com/paper/2003.08897/) (2020)
- [Typology of Risks of Generative Text-to-Image Models](https://mlsanity.com/paper/2307.05543/) (2023)
- [Evaluating the Robustness of Text-to-image Diffusion Models against Real-world Attacks](https://mlsanity.com/paper/2306.13103/) (2023)
- [Mitigating Inappropriateness in Image Generation: Can there be Value in Reflecting the World's Ugliness?](https://mlsanity.com/paper/2305.18398/) (2023)

## Built on by (most cited first, 30 of 66)

- [Direct Unlearning Optimization for Robust and Safe Text-to-Image Models](https://mlsanity.com/paper/2407.21035/) (2024)
- [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2411.13503/) (2024)
- [Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models](https://mlsanity.com/paper/2412.05934/) (2024)
- [Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Models](https://mlsanity.com/paper/2408.00523/) (2024)
- [Rethinking Data Protection in the (Generative) Artificial Intelligence Era](https://mlsanity.com/paper/2507.03034/) (2025)
- [PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models](https://mlsanity.com/paper/2501.03544/) (2025)
- [A Vision for Access Control in LLM-based Agent Systems](https://mlsanity.com/paper/2510.11108/) (2025)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [CHAI: Command Hijacking against embodied AI](https://mlsanity.com/paper/2510.00181/) (2025)
- [Towards Understanding Unsafe Video Generation](https://mlsanity.com/paper/2407.12581/) (2024)
- [DiffVax: Optimization-Free Image Immunization Against Diffusion-Based Editing](https://mlsanity.com/paper/2411.17957/) (2024)
- [Erased but Not Forgotten: How Backdoors Compromise Concept Erasure](https://mlsanity.com/paper/2504.21072/) (2025)
- [Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning](https://mlsanity.com/paper/2507.16302/) (2025)
- [PLA: Prompt Learning Attack against Text-to-Image Generative Models](https://mlsanity.com/paper/2508.03696/) (2025)
- [Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.09446/) (2025)
- [Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling](https://mlsanity.com/paper/2505.21074/) (2025)
- [ShieldDiff: Suppressing Sexual Content Generation from Diffusion Models through Reinforcement Learning](https://mlsanity.com/paper/2410.05309/) (2024)
- [Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimization](https://mlsanity.com/paper/2412.03876/) (2024)
- [Mitigating Sexual Content Generation via Embedding Distortion in Text-conditioned Diffusion Models](https://mlsanity.com/paper/2501.18877/) (2025)
- [Concept Steerers: Leveraging K-Sparse Autoencoders for Test-Time Controllable Generations](https://mlsanity.com/paper/2501.19066/) (2025)
- [SafeText: Safe Text-to-image Models via Aligning the Text Encoder](https://mlsanity.com/paper/2502.20623/) (2025)
- [GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Prompt](https://mlsanity.com/paper/2602.06258/) (2026)
- [Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models](https://mlsanity.com/paper/2503.12874/) (2025)
- [DyME: Dynamic Multi-Concept Erasure in Diffusion Models with Bi-Level Orthogonal LoRA Adaptation](https://mlsanity.com/paper/2509.21433/) (2025)
- [SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Models](https://mlsanity.com/paper/2510.05173/) (2025)
- [Dynamic Optimization and Safety Indicator Injection for Jailbreaking Text-to-Image Models with Multimodal Safety Filters](https://mlsanity.com/paper/2505.18979/) (2025)
- [DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modeling](https://mlsanity.com/paper/2507.16329/) (2025)
- [Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocols](https://mlsanity.com/paper/2507.19894/) (2025)
- [SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability](https://mlsanity.com/paper/2510.23960/) (2025)
- [ActErase: A Training-Free Paradigm for Precise Concept Erasure via Activation Redirection](https://mlsanity.com/paper/2601.00267/) (2026)

## Abstract (arXiv)

Text-to-image (T2I) models, such as Stable Diffusion, have exhibited remarkable performance in generating high-quality images from text descriptions in recent years. However, text-to-image models may be tricked into generating not-safe-for-work (NSFW) content, particularly in sexually explicit scenarios. Existing countermeasures mostly focus on filtering inappropriate inputs and outputs, or suppressing improper text embeddings, which can block sexually explicit content (e.g., naked) but may still be vulnerable to adversarial prompts -- inputs that appear innocent but are ill-intended. In this paper, we present SafeGen, a framework to mitigate sexual content generation by text-to-image models in a text-agnostic manner. The key idea is to eliminate explicit visual representations from the model regardless of the text input. In this way, the text-to-image model is resistant to adversarial prompts since such unsafe visual representations are obstructed from within. Extensive experiments conducted on four datasets and large-scale user studies demonstrate SafeGen's effectiveness in mitigating sexually explicit content generation while preserving the high-fidelity of benign images. SafeGen outperforms eight state-of-the-art baseline methods and achieves 99.4% sexual content removal performance. Furthermore, our constructed benchmark of adversarial prompts provides a basis for future development and evaluation of anti-NSFW-generation methods.