mlsanityData to 7 Oct 2026

HeadGuard: Selective Head Protection for Low-Bit VLM KV-Cache Quantization

arXiv 2609.35800 ↗First posted 18 Sep 2026cs.LG cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Low-bit key-value (KV) cache quantization saves storage but can sharply degrade vision-language model (VLM) accuracy. We introduce HeadGuard, a composable head-protection method that augments a base KV-cache quantizer with a fixed high-precision mask. Image-sensitivity and output-sensitivity scores select physical KV heads offline, with approximately 1/8 protected in the main experiments; their image keys and optionally values remain in bfloat16 (BF16), while the base quantizes unprotected image entries. Across eight VLMs, three base quantizers, and eight benchmarks (six discriminative and two generative), HeadGuard recovers a substantial fraction of lost accuracy on weaker quantizers, with the strongest gains for Qwen and InternVL. At 2 bits, the six-task discriminative mean over eight models rises from 0.436 to 0.580 on the weakest base; protection can also improve generated answers and caption fidelity to BF16 outputs. Mean accuracy gains persist across all three quantizers with both tested calibration datasets. Keys-only protection retains substantial recovery at lower modeled storage cost. Evaluated through simulated quantization, HeadGuard offers a composable way to improve low-bit VLM accuracy without replacing the underlying quantizer.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 29 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

26 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs5 Jun
  2. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More17 Feb
  3. 2025AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models25 Jan

Other references · 23

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2025Gemma 3 Technical Reportcited by 1,686
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
11 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  6. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  7. 2026Gemma 4 Technical Reportcited by 187
  8. 2024Pixtral 12Bcited by 174
  9. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  10. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cachecited by 4
  11. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theorycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
  2. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
  3. 2026Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation
  4. 2026Runtime-Certified Bounded-Error Quantized Attention
  5. 2026LQA: A Lightweight Quantized-Adaptive Framework for Vision-Language Models on the Edge
  6. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models
  7. 2026KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy
  8. 2026RoPE-Aware Bit Allocation for KV-Cache Quantization