# Attack Ensembles Expose a Safety-Utility Trade-off in Black-Box Guard Defenses Against Encoded VLM Jailbreaks

arXiv: https://arxiv.org/abs/2607.26574 · HTML: https://mlsanity.com/paper/2607.26574/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 72 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 27 of 27)

- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [Qwen3Guard Technical Report](https://mlsanity.com/paper/2510.14276/) (2025)
- [Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations](https://mlsanity.com/paper/2411.10414/) (2024)
- [Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency](https://mlsanity.com/paper/2501.04931/) (2025)
- [GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning](https://mlsanity.com/paper/2505.11049/) (2025)
- [Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey](https://mlsanity.com/paper/2411.09259/) (2024)
- [Jailbreaking Large Language Models with Symbolic Mathematics](https://mlsanity.com/paper/2409.11445/) (2024)
- [Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Search](https://mlsanity.com/paper/2602.22983/) (2026)
- [SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism](https://mlsanity.com/paper/2507.01513/) (2025)
- [Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models](https://mlsanity.com/paper/2505.16446/) (2025)
- [Cross-Modal Obfuscation for Jailbreak Attacks on Large Vision-Language Models](https://mlsanity.com/paper/2506.16760/) (2025)
- [Towards Robust Multimodal Large Language Models Against Jailbreak Attacks](https://mlsanity.com/paper/2502.00653/) (2025)
- [VRSA: Jailbreaking Multimodal Large Language Models through Visual Reasoning Sequential Attack](https://mlsanity.com/paper/2512.05853/) (2025)
- [The Great Contradiction Showdown: How Jailbreak and Stealth Wrestle in Vision-Language Models?](https://mlsanity.com/paper/2410.01438/) (2024)
- [Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift](https://mlsanity.com/paper/2603.17372/) (2026)
- [Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderation](https://mlsanity.com/paper/2604.06950/) (2026)
- [Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defense](https://mlsanity.com/paper/2601.03594/) (2026)
- [Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models](https://mlsanity.com/paper/2510.25179/) (2025)
- [Why Does Weak-OOD Help? A Further Step Towards Understanding Jailbreaking VLMs](https://mlsanity.com/paper/2511.08367/) (2025)
- [DefenSee: Dissecting Threat from Sight and Text -- A Multi-View Defensive Pipeline for Multi-modal Jailbreaks](https://mlsanity.com/paper/2512.01185/) (2025)
- [Text is All You Need for Vision-Language Model Jailbreaking](https://mlsanity.com/paper/2602.00420/) (2026)
- [Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection](https://mlsanity.com/paper/2604.09024/) (2026)
- [Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification](https://mlsanity.com/paper/2605.00326/) (2026)
- [When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness?](https://mlsanity.com/paper/2605.27932/) (2026)
- [DoubtProbe: Black-Box Jailbreak Defense via Structural Verification and Semantic Auditing](https://mlsanity.com/paper/2606.16527/) (2026)
- [Resolution Thresholds in VLM Detection of Harmful ASCII Art Across Construction Modes and Languages](https://mlsanity.com/paper/2606.29649/) (2026)

## Abstract (arXiv)

Safety classifiers ("guards") are the dominant black-box defense for vision-language models, yet a guard judges an input's surface form, not its meaning: a harmful request re-encoded as set theory, formal logic, a classical language, code, or text rendered inside an image slips past a guard that would block it in plain language - the decode gap. The standard fix is a preprocessor that recovers image content and decodes the encoding before the guard. We build one and evaluate it against an ensemble of eleven encoding attacks - six published implementations, one standard encoding baseline, one adapted and three author-constructed renders - counting a behavior as broken if any attack succeeds. Restoring a view the guard never had is what buys coverage - block rates on image renders go from exactly zero to 67-90% - and what it costs in benign traffic is set by the guard, not by the mechanism: one guard pays 9 benign blocking points for the same 70-point gain another pays 69 for. It still does not make the system safer: against an attacker free to choose among eleven encodings, closing one channel relocates the success rather than removing it, and no ensemble contrast for that step survives multiple-comparison correction. What does lower ensemble attack success is a reguard step that re-screens the recovered pre-decode surface, and it is the one every guard pays for: it raises benign over-refusal on all ten guard-target pairs, where restoring a single channel raises it on some and not others. Across the full guard x target x condition factorial, no configuration reaches an ensemble attack-success rate at or below 40% while holding benign over-refusal under 70%. That empty region is a property of the configurations we sample, not a bound on what recovery-based defenses can reach, and we breach its safety half ourselves.