mlsanityData to 7 Oct 2026

LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models

arXiv 2406.05113 ↗First posted 7 Jun 2024cs.CV cs.AI cs.LGcited by 49

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

This paper introduces LlavaGuard, a suite of VLM-based vision safeguards that address the critical need for reliable guardrails in the era of large-scale data and models. To this end, we establish a novel open framework, describing a customizable safety taxonomy, data preprocessing, augmentation, and training setup. For teaching a VLM safeguard on safety, we further create a multimodal safety dataset with high-quality human expert annotations, where each image is labeled with a safety rating, category, and rationale. We also employ advanced augmentations to support context-specific assessments. The resulting LlavaGuard models, ranging from 0.5B to 7B, serve as a versatile tool for evaluating the safety compliance of visual content against flexible policies. In comprehensive experiments, LlavaGuard outperforms both state-of-the-art safeguards and VLMs in accuracy and in flexibly handling different policies. Additionally, we demonstrate LlavaGuard's performance in two real-world applications: large-scale dataset annotation and moderation of text-to-image models. We make our entire framework, including the dataset, model weights, and training code.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 25 of its 45 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

25 of 45 references.

References · 25

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2018Datasheets for Datasetscited by 1,371
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
13 more
  1. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  2. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  3. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  4. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  5. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  6. 2023An Overview of Catastrophic AI Riskscited by 214
  7. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  8. 2023Into the LAIONs Den: Investigating Hate in Multimodal Datasetscited by 54
  9. 2023An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Modelscited by 24
  10. 2023Mitigating Inappropriateness in Image Generation: Can there be Value in Reflecting the World's Ugliness?cited by 16
  11. 2024Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help Youcited by 15
  12. 2023Distilling Adversarial Prompts from Safety Benchmarks: Report for the Adversarial Nibbler Challengecited by 6
  13. 2023Amplifying Limitations, Harms and Risks of Large Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 30.

49 papers.

Sort:

By area:

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2025PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Modelscited by 16
  4. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 10
  5. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Modelscited by 9
  6. 2025Systems Security Foundations for Agentic Computingcited by 9
  7. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  8. 2025AIN: The Arabic INclusive Large Multimodal Modelcited by 8
  9. 2025VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safetycited by 7
  10. 2025DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Modelscited by 6
  11. 2026Towards Policy-Adaptive Image Guardrail: Benchmark and Methodcited by 6
  12. 2025SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMscited by 5
37 more
  1. 2025EcoAlign: An Economically Rational Framework for Efficient LVLM Alignmentcited by 5
  2. 2025SafeMT: Multi-turn Safety for Multimodal Language Modelscited by 4
  3. 2025EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detectioncited by 3
  4. 2026Conditioned Activation Transport for T2I Safety Steeringcited by 3
  5. 2025SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generationcited by 2
  6. 2025SPQR: A Multi-Dimensional Benchmark for Safety Alignment under Benign Model Adaptationcited by 2
  7. 2025ProGuard: Towards Proactive Multimodal Safeguardcited by 2
  8. 2026BLM-Guard: Explainable Multimodal Ad Moderation with Chain-of-Thought and Policy-Aligned Rewardscited by 2
  9. 2026FlowGuard: Towards Lightweight In-Generation Safety Detection for Diffusion Models via Linear Latent Decodingcited by 2
  10. 2025LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Modelscited by 1
  11. 2025Agentic Moderation: Multi-Agent Design for Safer Vision-Language Modelscited by 1
  12. 2026RuleSafe-VL: Evaluating Rule-Conditioned Decision Reasoning in Vision-Language Content Moderationcited by 1
  13. 2026Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global Southcited by 1
  14. 2026EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agentscited by 1
  15. 2026Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generationcited by 1
  16. 2026SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safetycited by 0
  17. 2026EDGE-Shield: Efficient Denoising-staGE Shield for Violative Content Filtering via Scalable Reference-Based Matchingcited by 0
  18. 2026SenBen: Sensitive Scene Graphs for Explainable Content Moderationcited by 0
  19. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0
  20. 2026Certified Robustness under Heterogeneous Perturbations via Hybrid Randomized Smoothingcited by 0
  21. 2026SafeLens: Deliberate and Efficient Video Guardrails with Fast-and-Slow Screeningcited by 0
  22. 2026DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputscited by 0
  23. 2026No Safe Dose: How Training Data Drives Unsafe Image Generationcited by 0
  24. 2026Safe Alone, Unsafe Together: Safeguarding Against Implicit Toxicity When Benign Images Combinecited by 0
  25. 2026PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrailscited by 0
  26. 2026Automatic Hard Example Synthesis with Multi-Level Agentic Data Curationcited by 0
  27. 2026Harm is not Universal: Community-Specific Toxicity Detection is Urgently Neededcited by 0
  28. 2026Old Tricks, New Models: How Simple Image Transformations Break Modern AI-based Content Moderationcited by 0
  29. 2026TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Modelscited by 0
  30. 2026SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Modelscited by 0
  31. 2026Visual Compliance via Executable Safety Rule Entailmentcited by 0
  32. 2026Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Modelscited by 0
  33. 2026VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scalecited by 0
  34. 2026ThinkingGuard: Decoding Implicit Hazards via Step-by-Step Risk Attribution in Multimodal Large Language Modelscited by 0
  35. 2026It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at ε≤ 4/255cited by 0
  36. 2026ShieldCLIP: Selective Safety Alignment for Harmful Content Mitigation in Multimodal Foundation Modelscited by 0
  37. 2026When Scene Text Hijacks the Scene: Uncovering, Exploiting, and Mitigating Rendered-Text Semantic Leakage in Image Generation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations
  2. 2026Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMs
  3. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
  4. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
  5. 2024Safety Alignment for Vision Language Models
  6. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models
  7. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails
  8. 2025VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization