mlsanityData to 14 Oct 2026

SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations

arXiv 2412.06878 ↗First posted 9 Dec 2024cs.CV cs.LGcited by 38

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

With the rise of generative AI and rapid growth of high-quality video generation, video guardrails have become more crucial than ever to ensure safety and security across platforms. Current video guardrails, however, are either overly simplistic, relying on pure classification models trained on simple policies with limited unsafe categories, which lack detailed explanations, or prompting multimodal large language models (MLLMs) with long safety guidelines, which are inefficient and impractical for guardrailing real-world content. To bridge this gap, we propose SafeWatch, an efficient MLLM-based video guardrail model designed to follow customized safety policies and provide multi-label video guardrail outputs with content-specific explanations in a zero-shot manner. In particular, unlike traditional MLLM-based guardrails that encode all safety policies autoregressively, causing inefficiency and bias, SafeWatch uniquely encodes each policy chunk in parallel and eliminates their position bias such that all policies are attended simultaneously with equal importance. In addition, to improve efficiency and accuracy, SafeWatch incorporates a policy-aware visual token pruning algorithm that adaptively selects the most relevant video tokens for each policy, discarding noisy or irrelevant information. This allows for more focused, policy-compliant guardrail with significantly reduced computational overhead. Considering the limitations of existing video guardrail benchmarks, we propose SafeWatch-Bench, a large-scale video guardrail benchmark comprising over 2M videos spanning six safety categories which covers over 30 tasks to ensure a comprehensive coverage of all potential safety scenarios. SafeWatch outperforms SOTA by 28.2% on SafeWatch-Bench, 13.6% on benchmarks, cuts costs by 10%, and delivers top-tier explanations validated by LLM and human reviews.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 48 of its 60 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

48 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024T2Vs Meet VLMs: A Scalable Multimodal Dataset for Visual Harmfulness Recognition29 Sep

Other references · 47

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  9. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  12. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
35 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2022Red Teaming Language Models with Language Modelscited by 1,026
  3. 2022Token Merging: Your ViT But Fastercited by 1,021
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  6. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  7. 2018Real-world Anomaly Detection in Surveillance Videoscited by 577
  8. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  9. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
  10. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  11. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  12. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  13. 2020Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervisioncited by 238
  14. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  15. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 181
  16. 2020TransNet V2: An effective deep network architecture for fast shot transition detectioncited by 171
  17. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 158
  18. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  19. 2021UBnormal: New Benchmark for Supervised Open-Set Video Anomaly Detectioncited by 93
  20. 2024Harnessing Large Language Models for Training-free Video Anomaly Detectioncited by 87
  21. 2023Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokenscited by 71
  22. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  23. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  24. 2024Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLMcited by 64
  25. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
  26. 2020Localizing Anomalies from Weakly-Labeled Videoscited by 52
  27. 2023Hierarchical Spatio-temporal Decoupling for Text-to-Video Generationcited by 50
  28. 2024Hawk: Learning to Understand Open-World Video Anomaliescited by 49
  29. 2024Video Anomaly Detection and Explanation via Large Language Modelscited by 47
  30. 2022FakeSV: A Multimodal Benchmark with Rich Social Context for Fake News Detection on Short Video Platformscited by 41
  31. 2023AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detectioncited by 23
  32. 2024Can Editing LLMs Inject Harm?cited by 22
  33. 2023In-Contextual Gender Bias Suppression for Large Language Modelscited by 12
  34. 2022SoK: Content Moderation in Social Media, from Guidelines to Enforcement, and Research to Practicecited by 6
  35. 2023Malicious or Benign? Towards Effective Content Moderation for Children's Videoscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 22.

38 papers.

Sort:

By area:

  1. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 151
  2. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  3. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 34
  4. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  5. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  6. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 18
  7. 2025T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Modelcited by 17
  8. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 16
  9. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  10. 2025VidGuard-R1: AI-Generated Video Detection and Explanation via Reasoning MLLMs and RLcited by 15
  11. 2025Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMscited by 11
  12. 2025Debiasing Reward Models by Representation Learning with Guaranteescited by 6
26 more
  1. 2026Towards Policy-Adaptive Image Guardrail: Benchmark and Methodcited by 6
  2. 2025Failures to Surface Harmful Contents in Video Large Language Modelscited by 5
  3. 2025SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainabilitycited by 4
  4. 2025SPARK: Jailbreaking T2V Models by Synergistically Prompting Auditory and Recontextualized Knowledgecited by 4
  5. 2025ProGuard: Towards Proactive Multimodal Safeguardcited by 4
  6. 2026Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderationcited by 4
  7. 2026BLM-Guard: Explainable Multimodal Ad Moderation with Chain-of-Thought and Policy-Aligned Rewardscited by 3
  8. 2026Two Frames Matter: A Temporal Attack for Text-to-Video Model Jailbreakingcited by 3
  9. 2026AudioGuard: Toward Comprehensive Audio Safety Protection Across Diverse Threat Modelscited by 3
  10. 2026EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agentscited by 3
  11. 2025SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agentscited by 2
  12. 2026GuardReasoner-Omni: A Reasoning-based Multi-modal Guardrail for Text, Image, Video, and Audiocited by 2
  13. 2026TrajShield: Trajectory-Level Safety Mediation for Defending Text-to-Video Models Against Jailbreak Attackscited by 1
  14. 2026Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systemscited by 1
  15. 2026Pulling The REINS: Training-Free Safety Alignment of Video Diffusion Models via Representation Steeringcited by 1
  16. 2026EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guardscited by 1
  17. 2026Latent Space Probing for Adult Content Detection in Video Generative Modelscited by 0
  18. 2026SafeLens: Deliberate and Efficient Video Guardrails with Fast-and-Slow Screeningcited by 0
  19. 2026PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrailscited by 0
  20. 2026From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMscited by 0
  21. 2026SafeCA: Safe Cross-Attention Localization and Regulation for Text-to-Video Jailbreak Defensecited by 0
  22. 2026Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generationcited by 0
  23. 2026Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderationcited by 0
  24. 2026A Survey on Adversarial Attacks and Defenses for Diffusion Models Across Multiple Modalitiescited by 0
  25. 2026Trajectory Unlearning on LLM-based Agentscited by 0
  26. 2026Controllable Multi-label Video Safety Detection via Adaptive Tversky Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing
  2. 2025Guard Vector: Beyond English LLM Guardrails with Task-Vector Composition and Streaming-Aware Prefix SFT
  3. 2024LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models
  4. 2026LongGuard: Mechanistic Analysis and Training-Free Mitigation of Long-Context Failure in Safety Guardrails
  5. 2025OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoning
  6. 2025Protect: Towards Robust Guardrailing Stack for Trustworthy Enterprise LLM Systems
  7. 2026Triaging Threats to Specialized Guardrails
  8. 2026BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debate