# GuardT2I: Defending Text-to-Image Models from Adversarial Prompts

arXiv: https://arxiv.org/abs/2403.01446 · HTML: https://mlsanity.com/paper/2403.01446/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security.
- Builds on 42 of its 61 references that are in the map.
- First posted on arXiv 3 Mar 2024.

## Placed in

- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?](https://mlsanity.com/paper/2310.10012/) (2023)
- [Beam Search Strategies for Neural Machine Translation](https://mlsanity.com/paper/1702.01806/) (2017)
- [To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now](https://mlsanity.com/paper/2310.11868/) (2023)
- [MMA-Diffusion: MultiModal Attack on Diffusion Models](https://mlsanity.com/paper/2311.17516/) (2023)

## Built on by (most cited first, 30 of 62)

- [SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation](https://mlsanity.com/paper/2410.12761/) (2024)
- [Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents](https://mlsanity.com/paper/2411.09523/) (2024)
- [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2411.13503/) (2024)
- [Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Concepts](https://mlsanity.com/paper/2410.12777/) (2024)
- [Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2407.15861/) (2024)
- [Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey](https://mlsanity.com/paper/2408.03400/) (2024)
- [Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey](https://mlsanity.com/paper/2411.09259/) (2024)
- [Espresso: Robust Concept Filtering in Text-to-Image Models](https://mlsanity.com/paper/2404.19227/) (2024)
- [Concept Corrector: Erase concepts on the fly for text-to-image diffusion models](https://mlsanity.com/paper/2502.16368/) (2025)
- [AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models](https://mlsanity.com/paper/2410.21471/) (2024)
- [Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppression](https://mlsanity.com/paper/2505.19398/) (2025)
- [A Comprehensive Survey on Concept Erasure in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2502.14896/) (2025)
- [Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Models](https://mlsanity.com/paper/2504.11106/) (2025)
- [Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning](https://mlsanity.com/paper/2507.16302/) (2025)
- [PLA: Prompt Learning Attack against Text-to-Image Generative Models](https://mlsanity.com/paper/2508.03696/) (2025)
- [Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling](https://mlsanity.com/paper/2505.21074/) (2025)
- [AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Models](https://mlsanity.com/paper/2412.18123/) (2024)
- [Mitigating Sexual Content Generation via Embedding Distortion in Text-conditioned Diffusion Models](https://mlsanity.com/paper/2501.18877/) (2025)
- [TarPro: Targeted Protection against Malicious Image Editing](https://mlsanity.com/paper/2503.13994/) (2025)
- [Training-Free Safe Text Embedding Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2510.24012/) (2025)
- [SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Models](https://mlsanity.com/paper/2510.05173/) (2025)
- [Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control](https://mlsanity.com/paper/2503.18324/) (2025)
- [Dynamic Optimization and Safety Indicator Injection for Jailbreaking Text-to-Image Models with Multimodal Safety Filters](https://mlsanity.com/paper/2505.18979/) (2025)
- [Universally Unfiltered and Unseen:Input-Agnostic Multimodal Jailbreaks against Text-to-Image Model Safeguards](https://mlsanity.com/paper/2508.05658/) (2025)
- [PurifyGen: A Risk-Discrimination and Semantic-Purification Model for Safe Text-to-Image Generation](https://mlsanity.com/paper/2512.23546/) (2025)
- [Safe Text-to-Image Generation: Simply Sanitize the Prompt Embedding](https://mlsanity.com/paper/2411.10329/) (2024)
- [Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generation](https://mlsanity.com/paper/2412.07249/) (2024)
- [Antelope: Potent and Concealed Jailbreak Attack Strategy](https://mlsanity.com/paper/2412.08156/) (2024)
- [From Noise to Nuance: Advances in Deep Generative Image Models](https://mlsanity.com/paper/2412.09656/) (2024)
- [Wukong Framework for Not Safe For Work Detection in Text-to-Image systems](https://mlsanity.com/paper/2508.00591/) (2025)

## Abstract (arXiv)

Recent advancements in Text-to-Image (T2I) models have raised significant safety concerns about their potential misuse for generating inappropriate or Not-Safe-For-Work (NSFW) contents, despite existing countermeasures such as NSFW classifiers or model fine-tuning for inappropriate concept removal. Addressing this challenge, our study unveils GuardT2I, a novel moderation framework that adopts a generative approach to enhance T2I models' robustness against adversarial prompts. Instead of making a binary classification, GuardT2I utilizes a Large Language Model (LLM) to conditionally transform text guidance embeddings within the T2I models into natural language for effective adversarial prompt detection, without compromising the models' inherent performance. Our extensive experiments reveal that GuardT2I outperforms leading commercial solutions like OpenAI-Moderation and Microsoft Azure Moderator by a significant margin across diverse adversarial scenarios. Our framework is available at https://github.com/cure-lab/GuardT2I.