mlsanityData to 7 Oct 2026

MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models

arXiv 2603.02482 ↗First posted 3 Mar 2026cs.LG cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Safety evaluation of multimodal large language models requires tracking not only whether an attack succeeds, but also how the interaction unfolds across turns and input modalities. We present MUSE (Multimodal Unified Safety Evaluation), an open-source, browser-based, run-centric platform for multimodal safety evaluation. MUSE treats each attack run as the persistent unit of execution, inspection, and analysis, preserving its configuration, multi-turn trajectory, delivered modalities and media, target responses, and safety judgments. A five-level response taxonomy further distinguishes full Compliance from Partial Compliance and refusal behavior, yielding hard ASR, soft ASR, and gray-zone width (GZW). Across 11,700 evaluations on six multimodal LLMs, direct text-only requests yield only 3.1% macro hard ASR and 4.4% soft ASR, while iterative attack procedures are substantially more effective. Attack effectiveness also varies substantially with the attacker backbone. In contrast, Inter-Turn Modality Switching (ITMS), evaluated as a controlled delivery-modality probe, does not consistently increase attack success. These results demonstrate the value of run-centric, fine-grained evaluation for characterizing multimodal safety behavior beyond a single binary success metric.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 26 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

11 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs4 Jan
  2. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 9

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  6. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  7. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  8. 2024PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI Systemcited by 29
  9. 2025MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attackscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
  2. 2026MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
  3. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models
  4. 2025SafeMT: Multi-turn Safety for Multimodal Language Models
  5. 2025LLM-Safety Evaluations Lack Robustness
  6. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models
  7. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
  8. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues