# ShieldGemma 2: Robust and Tractable Image Content Moderation

arXiv: https://arxiv.org/abs/2504.01081 · HTML: https://mlsanity.com/paper/2504.01081/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [ShieldGemma: Generative AI Content Moderation Based on Gemma](https://mlsanity.com/paper/2407.21772/) (2024)
- [DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models](https://mlsanity.com/paper/2303.11681/) (2023)
- [DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models](https://mlsanity.com/paper/2308.06160/) (2023)
- [Latent Guard: a Safety Framework for Text-to-image Generation](https://mlsanity.com/paper/2404.08031/) (2024)
- [Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation](https://mlsanity.com/paper/2311.17216/) (2023)
- [UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images](https://mlsanity.com/paper/2405.03486/) (2024)
- [Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?](https://mlsanity.com/paper/2404.03411/) (2024)
- [Gen2Det: Generate to Detect](https://mlsanity.com/paper/2312.04566/) (2023)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Models](https://mlsanity.com/paper/2412.05538/) (2024)

## Built on by (most cited first, 29 of 29)

- [A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures](https://mlsanity.com/paper/2506.19676/) (2025)
- [Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models](https://mlsanity.com/paper/2507.13383/) (2025)
- [HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model](https://mlsanity.com/paper/2506.04704/) (2025)
- [Kelp: A Streaming Safeguard for Large Models via Latent Dynamics-Guided Risk Detection](https://mlsanity.com/paper/2510.09694/) (2025)
- [A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness](https://mlsanity.com/paper/2510.13890/) (2025)
- [FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderation](https://mlsanity.com/paper/2602.23636/) (2026)
- [SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning](https://mlsanity.com/paper/2606.22873/) (2026)
- [On the Adversarial Robustness of Multimodal LLM Judges](https://mlsanity.com/paper/2606.15608/) (2026)
- [Personalized Constitutionally-Aligned Agentic Superego: Secure AI Behavior Aligned to Diverse Human Values](https://mlsanity.com/paper/2506.13774/) (2025)
- [LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models](https://mlsanity.com/paper/2509.25896/) (2025)
- [MacPrompt: Maraconic-guided Jailbreak against Text-to-Image Models](https://mlsanity.com/paper/2601.07141/) (2026)
- [ExpGuard: LLM Content Moderation in Specialized Domains](https://mlsanity.com/paper/2603.02588/) (2026)
- [DataComp-VLM: Improved Open Datasets for Vision-Language Models](https://mlsanity.com/paper/2606.28551/) (2026)
- [Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?](https://mlsanity.com/paper/2510.21842/) (2025)
- [KidsNanny: A Two-Stage Multimodal Content Moderation Pipeline Integrating Visual Classification, Object Detection, OCR, and Contextual Reasoning for Child Safety](https://mlsanity.com/paper/2603.16181/) (2026)
- [UniSAFE: A Comprehensive Benchmark for Safety Evaluation of Unified Multimodal Models](https://mlsanity.com/paper/2603.17476/) (2026)
- [Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South](https://mlsanity.com/paper/2605.19190/) (2026)
- [SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing](https://mlsanity.com/paper/2606.29887/) (2026)
- [VLMs Can Aggregate Scattered Training Patches](https://mlsanity.com/paper/2506.03614/) (2025)
- [SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safety](https://mlsanity.com/paper/2603.02635/) (2026)
- [Structured Semantic Cloaking for Jailbreak Attacks on Large Language Models](https://mlsanity.com/paper/2603.16192/) (2026)
- [SenBen: Sensitive Scene Graphs for Explainable Content Moderation](https://mlsanity.com/paper/2604.08819/) (2026)
- [Boundary-targeted Membership Inference Attacks on Safety Classifiers](https://mlsanity.com/paper/2605.22373/) (2026)
- [D2-Monitor: Dynamic Safety Monitoring for Diffusion LLMs via Hesitation Signals](https://mlsanity.com/paper/2605.25893/) (2026)
- [No Safe Dose: How Training Data Drives Unsafe Image Generation](https://mlsanity.com/paper/2605.28137/) (2026)
- [Old Tricks, New Models: How Simple Image Transformations Break Modern AI-based Content Moderation](https://mlsanity.com/paper/2607.28187/) (2026)
- [ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions](https://mlsanity.com/paper/2608.10621/) (2026)
- [SafeAtlas-VL: Beyond Binary Multimodal Safety with Large-Scale Data and Guard Models](https://mlsanity.com/paper/2608.29098/) (2026)
- [Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderation](https://mlsanity.com/paper/2609.00206/) (2026)

## Abstract (arXiv)

We introduce ShieldGemma 2, a 4B parameter image content moderation model built on Gemma 3. This model provides robust safety risk predictions across the following key harm categories: Sexually Explicit, Violence & Gore, and Dangerous Content for synthetic images (e.g. output of any image generation model) and natural images (e.g. any image input to a Vision-Language Model). We evaluated on both internal and external benchmarks to demonstrate state-of-the-art performance compared to LlavaGuard citephelff2024llavaguard, GPT-4o mini citephurst2024gpt, and the base Gemma 3 model citepgemma2025 based on our policies. Additionally, we present a novel adversarial data generation pipeline which enables a controlled, diverse, and robust image generation. ShieldGemma 2 provides an open image moderation tool to advance multimodal safety and responsible AI development.