mlsanityData to 7 Oct 2026

AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models

arXiv 2501.15021 ↗First posted 25 Jan 2025cs.CLcited by 9

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) show remarkable performance in multimodal tasks. However, excessively long multimodal inputs lead to oversized Key-Value (KV) caches, resulting in significant memory consumption and I/O bottlenecks. Previous KV quantization methods for Large Language Models (LLMs) may alleviate these issues but overlook the attention saliency differences of multimodal tokens, resulting in suboptimal performance. In this paper, we investigate the attention-aware token saliency patterns in VLM and propose AKVQ-VL. AKVQ-VL leverages the proposed Text-Salient Attention (TSA) and Pivot-Token-Salient Attention (PSA) patterns to adaptively allocate bit budgets. Moreover, achieving extremely low-bit quantization requires effectively addressing outliers in KV tensors. AKVQ-VL utilizes the Walsh-Hadamard transform (WHT) to construct outlier-free KV caches, thereby reducing quantization difficulty. Evaluations of 2-bit quantization on 12 long-context and multimodal tasks demonstrate that AKVQ-VL maintains or even improves accuracy, outperforming LLM-oriented methods. AKVQ-VL can reduce peak memory usage by 2.13x, support up to 3.25x larger batch sizes and 2.46x throughput.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 21 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

20 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification11 Oct
  2. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
6 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  2. 2024Massive Activations in Large Language Modelscited by 277
  3. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  4. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  5. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  6. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 15
  2. 2026XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compressioncited by 6
  3. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  4. 2026SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipeliningcited by 3
  5. 2026SDFP: Speculative Decoding with FIT-Pruned Models for Training-Free and Plug-and-Play LLM Accelerationcited by 0
  6. 2026XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compressioncited by 0
  7. 2026WindowQuant: Mixed-Precision KV Cache Quantization based on Window-Level Similarity for VLMs Inference Optimizationcited by 0
  8. 2026KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancycited by 0
  9. 2026HeadGuard: Selective Head Protection for Low-Bit VLM KV-Cache Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
  2. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
  3. 2024A-VL: Adaptive Attention for Large Vision-Language Models
  4. 2026VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference
  5. 2026MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models
  6. 2025Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models
  7. 2026Towards Understanding Best Practices for Quantization of Vision-Language Models
  8. 2026LQA: A Lightweight Quantized-Adaptive Framework for Vision-Language Models on the Edge