mlsanityData to 14 Oct 2026

ShieldGemma 2: Robust and Tractable Image Content Moderation

arXiv 2504.01081 ↗First posted 1 Apr 2025cs.CV cs.CL eess.IVcited by 28

Not on the map. Why?

Abstract (arXiv)

We introduce ShieldGemma 2, a 4B parameter image content moderation model built on Gemma 3. This model provides robust safety risk predictions across the following key harm categories: Sexually Explicit, Violence & Gore, and Dangerous Content for synthetic images (e.g. output of any image generation model) and natural images (e.g. any image input to a Vision-Language Model). We evaluated on both internal and external benchmarks to demonstrate state-of-the-art performance compared to LlavaGuard citephelff2024llavaguard, GPT-4o mini citephurst2024gpt, and the base Gemma 3 model citepgemma2025 based on our policies. Additionally, we present a novel adversarial data generation pipeline which enables a controlled, diverse, and robust image generation. ShieldGemma 2 provides an open image moderation tool to advance multimodal safety and responsible AI development.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

25 of 33 references.

References · 25

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
13 more
  1. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  2. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  3. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  4. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  5. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  6. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 123
  7. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 71
  8. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  9. 2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 53
  10. 2024Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?cited by 34
  11. 2023Gen2Det: Generate to Detectcited by 12
  12. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  13. 2024Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 19.

28 papers.

Sort:

By area:

  1. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  2. 2025Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Modelscited by 11
  3. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Modelcited by 9
  4. 2025Kelp: A Streaming Safeguard for Large Models via Latent Dynamics-Guided Risk Detectioncited by 9
  5. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  6. 2026FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderationcited by 7
  7. 2026SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoningcited by 6
  8. 2026On the Adversarial Robustness of Multimodal LLM Judgescited by 3
  9. 2026MacPrompt: Maraconic-guided Jailbreak against Text-to-Image Modelscited by 2
  10. 2026ExpGuard: LLM Content Moderation in Specialized Domainscited by 2
  11. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Modelscited by 2
  12. 2025Personalized Constitutionally-Aligned Agentic Superego: Secure AI Behavior Aligned to Diverse Human Valuescited by 1
16 more
  1. 2025LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Modelscited by 1
  2. 2026KidsNanny: A Two-Stage Multimodal Content Moderation Pipeline Integrating Visual Classification, Object Detection, OCR, and Contextual Reasoning for Child Safetycited by 1
  3. 2026Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global Southcited by 1
  4. 2026SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailingcited by 1
  5. 2025VLMs Can Aggregate Scattered Training Patchescited by 0
  6. 2025Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?cited by 0
  7. 2026SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safetycited by 0
  8. 2026Structured Semantic Cloaking for Jailbreak Attacks on Large Language Modelscited by 0
  9. 2026SenBen: Sensitive Scene Graphs for Explainable Content Moderationcited by 0
  10. 2026Boundary-targeted Membership Inference Attacks on Safety Classifierscited by 0
  11. 2026D2-Monitor: Dynamic Safety Monitoring for Diffusion LLMs via Hesitation Signalscited by 0
  12. 2026No Safe Dose: How Training Data Drives Unsafe Image Generationcited by 0
  13. 2026Old Tricks, New Models: How Simple Image Transformations Break Modern AI-based Content Moderationcited by 0
  14. 2026ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributionscited by 0
  15. 2026SafeAtlas-VL: Beyond Binary Multimodal Safety with Large-Scale Data and Guard Modelscited by 0
  16. 2026Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Safe Synthetic Image Generation On the Web: A Multimodal Robust NSFW Defense and Million Scale Dataset
  2. 2026Harmful Content Generation in Text-to-Image Models: Capabilities and Moderation Limitations
  3. 2025PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models
  4. 2026SafeCtrl: Region-Aware Safety Control for Text-to-Image Diffusion via Detect-Then-Suppress
  5. 2026Safe Image Generation via Reinforcement Learning
  6. 2026ShieldCLIP: Selective Safety Alignment for Harmful Content Mitigation in Multimodal Foundation Models
  7. 2023Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models
  8. 2025Patronus: Safeguarding Text-to-Image Models against White-Box Adversaries