mlsanityData to 14 Oct 2026

High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models

arXiv 2512.21815 ↗First posted 26 Dec 2025cs.CV cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs) achieve remarkable performance but remain vulnerable to adversarial attacks. Entropy, as a measure of model uncertainty, is highly correlated with VLM reliability. While prior entropy-based attacks maximize uncertainty at all decoding steps, implicitly assuming that every token equally contributes to model instability, we reveal that a small fraction (around 20%) of high-entropy tokens, in the evaluated representative open-source VLMs with diverse architectures, concentrates a disproportionate share of adversarial influence during autoregressive generation. We demonstrate that concentrating adversarial perturbations on these high-entropy positions achieves comparable semantic degradation to global methods while optimizing fewer decoding positions. Additionally, across multiple representative VLMs, such attacks induce not only semantic drift but also a substantial unsafe subset (20-31%) under the current pipeline. Remarkably, since such vulnerable high-entropy tokens recur across architecturally diverse VLMs, attacks focused on them exhibit non-trivial transferability. Motivated by these findings, we design a simple Entropy-Guided Attack (EGA) that operationalizes sparse high-entropy targeting and extends it with a reusable token bank, yielding competitive attack success rates (93-95%) with a considerable harmful rate (30.2-38.6%) on the three representative open-source VLMs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 55 references that are in the map. First posted on arXiv 26 Dec 2025.

Builds on

39 of 55 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction25 Sep
  2. 2025Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models3 Aug
  3. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  4. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving27 Nov
  5. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks24 Nov
  6. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models9 Oct
  7. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  8. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 31

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  12. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
19 more
  1. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  2. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  3. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  4. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  5. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 178
  6. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  7. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
  8. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 31
  9. 2024VisionLLaMA: A Unified LLaMA Backbone for Vision Taskscited by 28
  10. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 27
  11. 2025LLM Jailbreak Detection for (Almost) Free!cited by 13
  12. 2019Identifying Model Weakness with Adversarial Examinercited by 12
  13. 2025Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCRcited by 12
  14. 2024Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imagingcited by 10
  15. 2025Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Ratecited by 10
  16. 2025Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Promptingcited by 8
  17. 2025Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognitioncited by 7
  18. 2025SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Modelscited by 5
  19. 2022Ask Before You Act: Generalising to Novel Environments by Asking Questionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression
  2. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  3. 2026Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
  4. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks
  5. 2026Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Models
  6. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  7. 2026Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models
  8. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models