mlsanityData to 14 Oct 2026

Visual Token Compression Enhances Robustness of MLLMs

arXiv 2607.22716 ↗First posted 21 Jul 2026cs.CV cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

In this paper, we show for the first time that visual token pruning enhances the robustness of Multimodal Large Language Models (MLLMs), mitigating vulnerabilities such as jailbreak attacks and hallucinations. Given that vision and language modalities cannot be perfectly aligned, the misaligned visual tokens might act as out-of-distribution (OOD) inputs, leading to unpredictable outputs and introducing potential vulnerabilities. Building on this insight, we aim to enhance model robustness against jailbreaks and hallucinations by reducing OOD visual tokens at robust-pruning layers, while also reducing inference cost as a side benefit. Specifically, we measure the distance between each visual token and the language feature space. Then, visual tokens with large distances are identified as OOD tokens, which can be iteratively pruned. To demonstrate the effectiveness of our method, we evaluate it on seven diverse popular benchmarks. Notably, our method yields an average improvement of 13.29% in defending jailbreak attacks, consistently achieves competitive performance in mitigating hallucinations, and maintains strong results on general datasets like MME.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 74 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

44 of 74 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression29 Jan
  2. 2025FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction25 Sep
  3. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks23 Nov
  4. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time9 Oct
  5. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Models21 Jul
  6. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  7. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  8. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
  9. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  10. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  11. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  12. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  8. 2018Object Hallucination in Image Captioningcited by 669
  9. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  10. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  11. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
20 more
  1. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  2. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  4. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  5. 2021Parametric Contrastive Learningcited by 217
  6. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  7. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  8. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  9. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 155
  10. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  11. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 139
  12. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  13. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 101
  14. 2018Sparse DNNs with Improved Adversarial Robustnesscited by 72
  15. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  16. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategycited by 44
  17. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameterscited by 25
  18. 2024Classes Are Not Equal: An Empirical Study on Image Recognition Fairnesscited by 10
  19. 2025Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustnesscited by 9
  20. 2024Typicalness-Aware Learning for Failure Detectioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Baseline-Relative Counterfactual Refinement for Bit-Aware Visual Token Communicationcited by 1
  2. 2026Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models
  2. 2026Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models
  3. 2026Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs
  4. 2026Understanding and Mitigating Token-Pruning-Induced Vulnerabilities in VLMs
  5. 2025Dynamic Token Reweighting for Robust Vision-Language Models
  6. 2025PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualization
  7. 2025JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering
  8. 2025PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMs