mlsanityData to 7 Oct 2026

UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images

arXiv 2405.03486 ↗First posted 6 May 2024cs.CR cs.CV cs.SIcited by 52

Not on the map. Why?

Abstract (arXiv)

With the advent of text-to-image models and concerns about their misuse, developers are increasingly relying on image safety classifiers to moderate their generated unsafe images. Yet, the performance of current image safety classifiers remains unknown for both real-world and AI-generated images. In this work, we propose UnsafeBench, a benchmarking framework that evaluates the effectiveness and robustness of image safety classifiers, with a particular focus on the impact of AI-generated images on their performance. First, we curate a large dataset of 10K real-world and AI-generated images that are annotated as safe or unsafe based on a set of 11 unsafe categories of images (sexual, violent, hateful, etc.). Then, we evaluate the effectiveness and robustness of five popular image safety classifiers, as well as three classifiers that are powered by general-purpose visual language models. Our assessment indicates that existing image safety classifiers are not comprehensive and effective enough to mitigate the multifaceted problem of unsafe images. Also, there exists a distribution shift between real-world and AI-generated images in image qualities, styles, and layouts, leading to degraded effectiveness and robustness. Motivated by these findings, we build a comprehensive image moderation tool called PerspectiveVision, which improves the effectiveness and robustness of existing classifiers, especially on AI-generated images. UnsafeBench and PerspectiveVision can aid the research community in better understanding the landscape of image safety classification in the era of generative AI.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 39 of its 68 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

39 of 68 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
27 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2016Universal adversarial perturbationscited by 774
  3. 2023Erasing Concepts from Diffusion Modelscited by 530
  4. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  5. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  6. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  7. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
  8. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 239
  9. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  10. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
  11. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 161
  12. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  13. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  14. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  15. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  16. 2021Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hatecited by 50
  17. 2019Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderationcited by 29
  18. 2018A Quantitative Approach to Understanding Online Antisemitismcited by 26
  19. 2022On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learningcited by 26
  20. 2023Stereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Modelscited by 26
  21. 2023A Friendly Face: Do Text-to-Image Systems Rely on Stereotypes when the Input is Under-Specified?cited by 25
  22. 2017Protest Activity Detection and Perceived Violence Estimation from Social Media Imagescited by 20
  23. 2023On the Proactive Generation of Unsafe Images From Text-To-Image Models Using Benign Promptscited by 14
  24. 2022Understanding and Detecting Hateful Content using Contrastive Learningcited by 13
  25. 2024Moderating Illicit Online Image Promotion for Unsafe User-Generated Content Games Using Large Vision-Language Modelscited by 13
  26. 2023Modeling the Trade-off of Privacy Preservation and Activity Recognition on Low-Resolution Imagescited by 6
  27. 2023Distilling Adversarial Prompts from Safety Benchmarks: Report for the Adversarial Nibbler Challengecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 25; 2026: 18.

52 papers.

Sort:

By area:

  1. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  2. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 59
  3. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  4. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 42
  5. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  6. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 27
  7. 2025ShieldGemma 2: Robust and Tractable Image Content Moderationcited by 27
  8. 2024MLLM-as-a-Judge for Image Safety without Human Labelingcited by 23
  9. 2025HateBench: Benchmarking Hate Speech Detectors on LLM-Generated Content and Hate Campaignscited by 10
  10. 2025Multimodal LLMs as Customized Reward Models for Text-to-Image Generationcited by 9
  11. 2024Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimizationcited by 7
  12. 2025VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safetycited by 7
38 more
  1. 2025RunawayEvil: Jailbreaking the Image-to-Video Generative Modelscited by 7
  2. 2025Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusionscited by 6
  3. 2026Towards Policy-Adaptive Image Guardrail: Benchmark and Methodcited by 6
  4. 2024Class-RAG: Real-Time Content Moderation with Retrieval Augmented Generationcited by 5
  5. 2025DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modelingcited by 5
  6. 2025OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoningcited by 5
  7. 2025Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalitiescited by 4
  8. 2025SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainabilitycited by 4
  9. 2026SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoningcited by 4
  10. 2024From Noise to Nuance: Advances in Deep Generative Image Modelscited by 3
  11. 2025LR0.FM: Low-Res Benchmark and Improving Robustness for Zero-Shot Classification in Foundation Modelscited by 3
  12. 2026SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generationcited by 3
  13. 2026On the Adversarial Robustness of Multimodal LLM Judgescited by 3
  14. 2024HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memescited by 2
  15. 2024Gen-AI for User Safety: A Surveycited by 2
  16. 2025Immunizing Images from Text to Image Editing via Adversarial Cross-Attentioncited by 2
  17. 2025SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generationcited by 2
  18. 2025SafeEditor: Unified MLLM for Efficient Post-hoc T2I Safety Editingcited by 2
  19. 2025SPQR: A Multi-Dimensional Benchmark for Safety Alignment under Benign Model Adaptationcited by 2
  20. 2025ProGuard: Towards Proactive Multimodal Safeguardcited by 2
  21. 2026Selective Fine-Tuning for Targeted and Robust Concept Unlearningcited by 2
  22. 2026BLM-Guard: Explainable Multimodal Ad Moderation with Chain-of-Thought and Policy-Aligned Rewardscited by 2
  23. 2025VModA: An Effective Framework for Adaptive NSFW Image Moderationcited by 1
  24. 2025Customize Multi-modal RAI Guardrails with Precedent-based predictionscited by 1
  25. 2025LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Modelscited by 1
  26. 2025CompAgent: An Agentic Framework for Visual Compliance Verificationcited by 1
  27. 2026Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judgecited by 1
  28. 2026VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attackscited by 1
  29. 2026Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generationcited by 1
  30. 2025SP-Guard: Selective Prompt-adaptive Guidance for Safe Text-to-Image Generationcited by 0
  31. 2025A Multi-Perspective Benchmark and Moderation Model for Evaluating Safety and Adversarial Robustnesscited by 0
  32. 2026Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judgecited by 0
  33. 2026SenBen: Sensitive Scene Graphs for Explainable Content Moderationcited by 0
  34. 2026When Safe Concepts Become Unsafe: Multi-Concept Compositional Vulnerabilities in Text-to-Image Modelscited by 0
  35. 2026On the Generation and Mitigation of Harmful Geometry in Image-to-3D Modelscited by 0
  36. 2026Robust and Generalizable Safety Steering for Text-to-Image Diffusion Transformerscited by 0
  37. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0
  38. 2026Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessmentcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model
  2. 2026AI-generated Images Challenge Visual Trust in High-risk Scenarios
  3. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs
  4. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
  5. 2024Image Safeguarding: Reasoning with Conditional Vision Language Model and Obfuscating Unsafe Content Counterfactually
  6. 2025ELITE: Enhanced Language-Image Toxicity Evaluation for Safety
  7. 2025Safe Vision-Language Models via Unsafe Weights Manipulation
  8. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models