mlsanityData to 14 Oct 2026

Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs

arXiv 2605.23157 ↗First posted 22 May 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

The attack surface of a multimodal large language model (MLLM) is language-dependent in ways that reveal the mechanistic structure of alignment failures. We present the first systematic cross-lingual, multimodal red-teaming study comparing jailbreak vulnerability in US English (en-US) and Mexican Spanish (es-MX) across four frontier MLLMs: Claude Sonnet 4.5, GPT-5, Pixtral Large, and Qwen Omni. Using a fixed adversarial benchmark of 363 diverse prompt scenarios administered in text-only and multimodal conditions, we collected 52,272 harm ratings and binary attack success judgements from matched panels of nine native-speaker annotators per language group. Our central finding is that language does not scale vulnerability uniformly. Bayesian mixed-effects analyses reveal that linguistic framing attacks such as role-play become substantially less effective under Spanish prompting, while visually explicit multimodal attacks become more effective, which directly implicates the prompt-language interface rather than global annotator leniency. This dissociation indicates that linguistic and visual alignment failures operate through distinct mechanisms, and that switching language is sufficient to expose that separation. The practical consequence is that safety rankings are not preserved across languages. Qwen Omni overtakes Pixtral Large as the most vulnerable model among es-MX participants, a rank reversal no scalar correction of English-condition scores could recover, and absolute attack success rates have declined across model generations without closing the gaps between them. These findings demonstrate that safety evaluation frameworks treating language and modality as independent dimensions fundamentally misspecify the attack surface of globally deployed MLLMs, and must be redesigned accordingly.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 40 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

24 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Red Teaming Multimodal Language Models: Evaluating Harm Across Prompt Modalities and Models18 Sep
  2. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models2 Jun
  3. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreaking15 Feb
  4. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar

Other references · 20

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  3. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  4. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  5. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  6. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  7. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  8. 2023Generative Language Models Exhibit Social Identity Biasescited by 111
  9. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 94
  10. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  11. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  12. 2024Query-Based Adversarial Prompt Generationcited by 62
8 more
  1. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
  2. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  3. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 39
  4. 2025Large Reasoning Models Are Autonomous Jailbreak Agentscited by 13
  5. 2025Enhancing Jailbreak Attacks on LLMs via Persona Promptscited by 11
  6. 2025Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMscited by 10
  7. 2025HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humorcited by 6
  8. 2025StyleBreak: Revealing Alignment Vulnerabilities in Large Audio-Language Models via Style-Aware Audio Jailbreakcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Models
  2. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
  3. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking
  4. 2025Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models
  5. 2025Say It Differently: Linguistic Styles as Jailbreak Vectors
  6. 2026MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models
  7. 2026MMJailBench: A Factorized Benchmark for Disentangling Multimodal Jailbreak Vulnerabilities
  8. 2026IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks