# T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model arXiv: https://arxiv.org/abs/2510.22300 · HTML: https://mlsanity.com/paper/2510.22300/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 41 references that are in the map. - First posted on arXiv 25 Oct 2025. ## Builds on (most cited first, 23 of 23) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023) - [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024) - [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024) - [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023) - [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025) - [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022) - [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023) - [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024) - [MMA-Diffusion: MultiModal Attack on Diffusion Models](https://mlsanity.com/paper/2311.17516/) (2023) - [Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?](https://mlsanity.com/paper/2202.06675/) (2022) - [SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation](https://mlsanity.com/paper/2410.12761/) (2024) - [Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations](https://mlsanity.com/paper/2411.10414/) (2024) - [CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion](https://mlsanity.com/paper/2403.05121/) (2024) - [T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models](https://mlsanity.com/paper/2407.05965/) (2024) - [T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation](https://mlsanity.com/paper/2501.12612/) (2025) - [SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations](https://mlsanity.com/paper/2412.06878/) (2024) - [SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models](https://mlsanity.com/paper/2503.07392/) (2025) - [SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset](https://mlsanity.com/paper/2406.14477/) (2024) - [Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2407.15861/) (2024) - [MLLM-as-a-Judge for Image Safety without Human Labeling](https://mlsanity.com/paper/2501.00192/) (2024) - [Generative AI in EU Law: Liability, Privacy, Intellectual Property, and Cybersecurity](https://mlsanity.com/paper/2401.07348/) (2024) ## Built on by (most cited first, 17 of 17) - [Conditioned Activation Transport for T2I Safety Steering](https://mlsanity.com/paper/2603.03163/) (2026) - [On the Adversarial Robustness of Multimodal LLM Judges](https://mlsanity.com/paper/2606.15608/) (2026) - [Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Models](https://mlsanity.com/paper/2603.27332/) (2026) - [GEM: Geometric Erasure by Contrastive Velocity Matching in Rectified Flows](https://mlsanity.com/paper/2606.00140/) (2026) - [What Concepts Lie Within? Detecting and Suppressing Risky Content in Diffusion Transformers](https://mlsanity.com/paper/2605.10180/) (2026) - [Obliviate: Erasing Concepts from Autoregressive Image Generation Models](https://mlsanity.com/paper/2606.28643/) (2026) - [When Safe Concepts Become Unsafe: Multi-Concept Compositional Vulnerabilities in Text-to-Image Models](https://mlsanity.com/paper/2604.15967/) (2026) - [Benign Inputs, Harmful Outputs: Cross-Modal Jailbreaking via Distributed Semantic Recomposition](https://mlsanity.com/paper/2606.01837/) (2026) - [PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation](https://mlsanity.com/paper/2606.24081/) (2026) - [Evaluating Intellectual Property Guardrails of Generative Image Models: A Technical Report](https://mlsanity.com/paper/2607.02582/) (2026) - [Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models](https://mlsanity.com/paper/2607.17779/) (2026) - [Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generation](https://mlsanity.com/paper/2608.26535/) (2026) - [SafeAtlas-VL: Beyond Binary Multimodal Safety with Large-Scale Data and Guard Models](https://mlsanity.com/paper/2608.29098/) (2026) - [Not Safe for All: Auditing the Dialect Penalty in Text-to-Image Safety Pipelines](https://mlsanity.com/paper/2608.29589/) (2026) - [AV-SafetyBench: A Safety Benchmark for Text-to-Audio-Video Generation](https://mlsanity.com/paper/2609.06991/) (2026) - [Steering Fields: Adaptive Vector Fields for Safe Image Generation and Beyond](https://mlsanity.com/paper/2609.39573/) (2026) - [UnifiedAttack: Evaluating the Safety of Large Multimodal Models in Synergistic Harmful Image-Text Generation](https://mlsanity.com/paper/2610.00341/) (2026) ## Abstract (arXiv) Using risky text prompts, such as pornography and violent prompts, to test the safety of text-to-image (T2I) models is a critical task. However, existing risky prompt datasets are limited in three key areas: 1) limited risky categories, 2) coarse-grained annotation, and 3) low effectiveness. To address these limitations, we introduce T2I-RiskyPrompt, a comprehensive benchmark designed for evaluating safety-related tasks in T2I models. Specifically, we first develop a hierarchical risk taxonomy, which consists of 6 primary categories and 14 fine-grained subcategories. Building upon this taxonomy, we construct a pipeline to collect and annotate risky prompts. Finally, we obtain 6,432 effective risky prompts, where each prompt is annotated with both hierarchical category labels and detailed risk reasons. Moreover, to facilitate the evaluation, we propose a reason-driven risky image detection method that explicitly aligns the MLLM with safety annotations. Based on T2I-RiskyPrompt, we conduct a comprehensive evaluation of eight T2I models, nine defense methods, five safety filters, and five attack strategies, offering nine key insights into the strengths and limitations of T2I model safety. Finally, we discuss potential applications of T2I-RiskyPrompt across various research fields. The dataset and code are provided in https://github.com/datar001/T2I-RiskyPrompt.