mlsanityData to 14 Oct 2026

Dynamic Token Reweighting for Robust Vision-Language Models

arXiv 2505.17132 ↗First posted 22 May 2025cs.CV cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Large vision-language models (VLMs) are highly vulnerable to multimodal jailbreak attacks that exploit visual-textual interactions to bypass safety guardrails. In this paper, we present DTR, a novel inference-time defense that mitigates multimodal jailbreak attacks through optimizing the model's key-value (KV) caches. Rather than relying on curated safety-specific data or costly image-to-text conversion, we introduce a new formulation of the safety-relevant distributional shift induced by the visual modality. This formulation enables DTR to dynamically adjust visual token weights, minimizing the impact of adversarial visual inputs while preserving the model's general capabilities and inference efficiency. Extensive evaluation across diverse VLMs and attack benchmarks demonstrates that DTR outperforms existing defenses in both attack robustness and benign-task performance, marking the first successful application of KV cache optimization for safety enhancement in multimodal foundation models. The code for replicating DTR is available at: https://github.com/TanqiuJiang/DTR.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 49 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

42 of 49 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States20 Feb
  2. 2025Understanding and Rectifying Safety Perception Distortion in VLMs18 Feb
  3. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment27 Nov
  4. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models11 Oct
  5. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time9 Oct
  6. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration17 Sep
  7. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking21 Jun
  8. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  9. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  10. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  11. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  12. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
4 more
  1. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  2. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  3. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
14 more
  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  3. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  5. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  6. 2024A StrongREJECT for Empty Jailbreakscited by 385
  7. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  8. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  9. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  10. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 118
  11. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  12. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 54
  13. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  14. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models
  2. 2025Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense
  3. 2025VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models
  4. 2026Visual Token Compression Enhances Robustness of MLLMs
  5. 2026When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness?
  6. 2026Jailbreaks on Vision Language Model via Multimodal Reasoning
  7. 2025TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models
  8. 2025SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models