# PromptSafe: Gated Prompt Tuning for Safe Text-to-Image Generation

arXiv: https://arxiv.org/abs/2508.01272 · HTML: https://mlsanity.com/paper/2508.01272/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 2 Aug 2025.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [MMA-Diffusion: MultiModal Attack on Diffusion Models](https://mlsanity.com/paper/2311.17516/) (2023)
- [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://mlsanity.com/paper/2305.12082/) (2023)
- [Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models](https://mlsanity.com/paper/2305.13873/) (2023)
- [Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?](https://mlsanity.com/paper/2202.06675/) (2022)
- [Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical World](https://mlsanity.com/paper/2103.01050/) (2021)
- [BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning](https://mlsanity.com/paper/2311.12075/) (2023)
- [RobustART: Benchmarking Robustness on Architecture Design and Training Techniques](https://mlsanity.com/paper/2109.05211/) (2021)
- [SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models](https://mlsanity.com/paper/2404.06666/) (2024)
- [Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Models](https://mlsanity.com/paper/2502.11054/) (2025)
- [Training Robust Deep Neural Networks via Adversarial Noise Propagation](https://mlsanity.com/paper/1909.09034/) (2019)
- [X-Adv: Physical Adversarial Object Attacks against X-ray Prohibited Item Detection](https://mlsanity.com/paper/2302.09491/) (2023)
- [Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift](https://mlsanity.com/paper/2406.18844/) (2024)
- [A Comprehensive Evaluation Framework for Deep Model Robustness](https://mlsanity.com/paper/2101.09617/) (2021)
- [Harnessing Perceptual Adversarial Patches for Crowd Counting](https://mlsanity.com/paper/2109.07986/) (2021)
- [T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models](https://mlsanity.com/paper/2504.15512/) (2025)
- [PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models](https://mlsanity.com/paper/2501.03544/) (2025)
- [Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark](https://mlsanity.com/paper/2411.15488/) (2024)
- [Harnessing LLM to Attack LLM-Guarded Text-to-Image Models](https://mlsanity.com/paper/2312.07130/) (2023)
- [CogMorph: Cognitive Morphing Attacks for Text-to-Image Models](https://mlsanity.com/paper/2501.11815/) (2025)
- [SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMs](https://mlsanity.com/paper/2506.04743/) (2025)
- [AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Models](https://mlsanity.com/paper/2412.18123/) (2024)
- [Mitigating Sexual Content Generation via Embedding Distortion in Text-conditioned Diffusion Models](https://mlsanity.com/paper/2501.18877/) (2025)
- [SafeText: Safe Text-to-image Models via Aligning the Text Encoder](https://mlsanity.com/paper/2502.20623/) (2025)

## Built on by (most cited first, 2 of 2)

- [Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.06837/) (2025)
- [DiSCO: Defending text-to-image generation through distribution-guided contrastive prompt optimization](https://mlsanity.com/paper/2608.17067/) (2026)

## Abstract (arXiv)

Text-to-image (T2I) models have demonstrated remarkable generative capabilities but remain vulnerable to producing not-safe-for-work (NSFW) content, such as violent or explicit imagery. While recent moderation efforts have introduced soft prompt-guided tuning by appending defensive tokens to the input, these approaches often rely on large-scale curated image-text datasets and apply static, one-size-fits-all defenses at inference time. However, this results not only in high computational cost and degraded benign image quality, but also in limited adaptability to the diverse and nuanced safety requirements of real-world prompts. To address these challenges, we propose PromptSafe, a gated prompt tuning framework that combines a lightweight, text-only supervised soft embedding with an inference-time gated control network. Instead of training on expensive image-text datasets, we first rewrite unsafe prompts into semantically aligned but safe alternatives using an LLM, constructing an efficient text-only training corpus. Based on this, we optimize a universal soft prompt that repels unsafe and attracts safe embeddings during the diffusion denoising process. To avoid over-suppressing benign prompts, we introduce a gated mechanism that adaptively adjusts the defensive strength based on estimated prompt toxicity, thereby aligning defense intensity with prompt risk and ensuring strong protection for harmful inputs while preserving benign generation quality. Extensive experiments across multiple benchmarks and T2I models show that PromptSafe achieves a SOTA unsafe generation rate (2.36%), while preserving high benign fidelity. Furthermore, PromptSafe demonstrates strong generalization to unseen harmful categories, robust transferability across diffusion model architectures, and resilience under adaptive adversarial attacks, highlighting its practical value for safe and scalable deployment.