# Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models

arXiv: https://arxiv.org/abs/2510.25179 · HTML: https://mlsanity.com/paper/2510.25179/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 30 of its 32 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models](https://mlsanity.com/paper/2307.14539/) (2023)
- [ShieldGemma: Generative AI Content Moderation Based on Gemma](https://mlsanity.com/paper/2407.21772/) (2024)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models](https://mlsanity.com/paper/2402.02207/) (2024)
- [Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations](https://mlsanity.com/paper/2411.10414/) (2024)
- [AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting](https://mlsanity.com/paper/2403.09513/) (2024)
- [SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model](https://mlsanity.com/paper/2406.12030/) (2024)
- [ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time](https://mlsanity.com/paper/2410.06625/) (2024)
- [LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models](https://mlsanity.com/paper/2406.05113/) (2024)
- [PAD: Patch-Agnostic Defense against Adversarial Patch Attacks](https://mlsanity.com/paper/2404.16452/) (2024)
- [Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model](https://mlsanity.com/paper/2406.15279/) (2024)
- [BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks](https://mlsanity.com/paper/2410.20971/) (2024)
- [The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense](https://mlsanity.com/paper/2411.08410/) (2024)
- [FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowcharts](https://mlsanity.com/paper/2502.21059/) (2025)
- [Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training](https://mlsanity.com/paper/2502.11455/) (2025)
- [Image-to-Text Logic Jailbreak: Your Imagination can Help You Do Anything](https://mlsanity.com/paper/2407.02534/) (2024)
- [Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack](https://mlsanity.com/paper/2505.21967/) (2025)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs](https://mlsanity.com/paper/2510.13190/) (2025)
- [Effective Black-Box Multi-Faceted Attacks Breach Vision Large Language Model Guardrails](https://mlsanity.com/paper/2502.05772/) (2025)

## Built on by (most cited first, 2 of 2)

- [DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responses](https://mlsanity.com/paper/2512.02282/) (2025)
- [Attack Ensembles Expose a Safety-Utility Trade-off in Black-Box Guard Defenses Against Encoded VLM Jailbreaks](https://mlsanity.com/paper/2607.26574/) (2026)

## Abstract (arXiv)

Agentic methods have emerged as a powerful and autonomous paradigm that enhances reasoning, collaboration, and adaptive control, enabling systems to coordinate and independently solve complex tasks. We extend this paradigm to safety alignment by introducing Agentic Moderation, a model-agnostic framework that leverages specialised agents to defend multimodal systems against jailbreak attacks. Unlike prior approaches that apply as a static layer over inputs or outputs and provide only binary classifications (safe or unsafe), our method integrates dynamic, cooperative agents, including Shield, Responder, Evaluator, and Reflector, to achieve context-aware and interpretable moderation. Extensive experiments across five datasets and four representative Large Vision-Language Models (LVLMs) demonstrate that our approach reduces the Attack Success Rate (ASR) by 7-19%, maintains a stable Non-Following Rate (NF), and improves the Refusal Rate (RR) by 4-20%, achieving robust, interpretable, and well-balanced safety performance. By harnessing the flexibility and reasoning capacity of agentic architectures, Agentic Moderation provides modular, scalable, and fine-grained safety enforcement, highlighting the broader potential of agentic systems as a foundation for automated safety governance.