mlsanityData to 14 Oct 2026

Break the Brake, Not the Wheel: Untargeted Jailbreak via Entropy Maximization

arXiv 2605.10764 ↗First posted 11 May 2026cs.CV cs.AIcited by 0

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Recent studies show that gradient-based universal image jailbreaks on vision-language models (VLMs) exhibit little or no cross-model transferability, casting doubt on the feasibility of transferable multimodal jailbreaks. We revisit this conclusion under a strictly untargeted threat model without enforcing a fixed prefix or response pattern. Our preliminary experiment reveals that refusal behavior concentrates at high-entropy tokens during autoregressive decoding, and non-refusal tokens already carry substantial probability mass among the top-ranked candidates before attack. Motivated by this finding, we propose Untargeted Jailbreak via Entropy Maximization(UJEM)-KL, a lightweight attack that maximizes entropy at these decision tokens to flip refusal outcomes, while stabilizing the remaining low-entropy positions to preserve output quality. Across three VLMs and two safety benchmarks, UJEM-KL achieves competitive white-box attack success rates and consistently improves transferability, while remaining effective under representative defenses. Our experimental results indicate that the limited transferability primarily stems from overly constrained optimization objectives.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 41 of its 51 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

41 of 51 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction25 Sep
  2. 2025JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation24 Sep
  3. 2025Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models3 Aug
  4. 2025Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack28 May
  5. 2024Gradient-based Jailbreak Images for Multimodal Fusion Models4 Oct
  6. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Models21 Jul
  7. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
21 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  3. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  4. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  5. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  6. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  7. 2023Are aligned neural networks adversarially aligned?cited by 315
  8. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  9. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  10. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  11. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 69
  12. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  13. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 27
  14. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 25
  15. 2021Closer Look at the Transferability of Adversarial Examples: How They Fool Different Models Differentlycited by 22
  16. 2025Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous Constraintscited by 11
  17. 2025R-TOFU: Unlearning in Large Reasoning Modelscited by 9
  18. 2025Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Promptingcited by 8
  19. 2025Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognitioncited by 7
  20. 2025SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Modelscited by 5
  21. 2025NonTextual Target Attackcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models
  2. 2024The Great Contradiction Showdown: How Jailbreak and Stealth Wrestle in Vision-Language Models?
  3. 2025Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models
  4. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  5. 2026Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models
  6. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation
  7. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks
  8. 2026PAST2HARM: A Simple Adaptive Past Tense Attack for Jailbreaking Multimodal AI