mlsanityData to 14 Oct 2026

MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs

arXiv 2609.20850 ↗First posted 8 Aug 2026cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) show remarkable advancements, their cross-modal capabilities introduce complex vulnerabilities that easily bypass unimodal filters. Existing benchmarks lack fine-grained intent-related annotations and rely on unidimensional metrics, hindering comprehensive robustness evaluation. To address this, we propose MME-Safety, a rigorously verified benchmark featuring a unique four-dimensional annotation schema that categorizes risk scenarios, harm severity, and modality-specific stealth levels. Furthermore, we introduce a hierarchical evaluation framework to assess fundamental response reliability, actual risk exposure, and the structural integrity of defensive behaviors. Extensive zero-shot evaluations across 17 state-of-the-art MLLMs provide a comprehensive safety profile of current multimodal systems. Our analysis systematically investigates cross-modal input configurations and uncovers safety implications associated with Chain-of-Thought (CoT) reasoning. These multifaceted findings underscore the urgent need for robust, reasoning-aware safety alignment in the multimodal landscape.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 13 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

8 of 13 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025VisCRA: A Visual Chain Reasoning Attack for Jailbreaking Multimodal Large Language Models26 May
  2. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safety29 Nov
  3. 2024Multimodal Situational Safety8 Oct

Other references · 5

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  2. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  3. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  4. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 11
  5. 2025When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignmentcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
  2. 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences
  3. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
  4. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models
  5. 2025When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models
  6. 2026When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Life
  7. 2026MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models
  8. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models