mlsanityData to 14 Oct 2026

WindowQuant: Mixed-Precision KV Cache Quantization based on Window-Level Similarity for VLMs Inference Optimization

arXiv 2605.02262 ↗First posted 4 May 2026cs.CV cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Recently, video language models (VLMs) have been applied in various fields. However, the visual token sequence of the VLM is too long, which may cause intolerant inference latency and GPU memory usage. Existing methods propose mixed-precision quantization to the key-value (KV) cache in VLMs based on token granularity, which is time-consuming in the search process and hardware inefficient during computation. This paper introduces a novel approach called WindowQuant, which employs window-adaptive mixed-precision quantization to optimize the KV cache. WindowQuant consists of two modules: window-level quantization search and window-level KV cache computation. Window-level quantization search quickly determines the optimal bit-width configuration of the KV cache windows based on the similarity scores between the corresponding visual token windows and the text prompt, maintaining the model accuracy. Furthermore, window-level KV cache computation reorders the KV cache windows before quantization, avoiding the hardware inefficiency caused by mixed-precision quantization in inference computation. Extensive experiments demonstrate that WindowQuant outperforms state-of-the-art VLM models and KV cache quantization methods on various datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 50 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

39 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  2. 2024QAQ: Quality Adaptive Quantization for LLM KV Cache7 Mar
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  4. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
23 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  12. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  13. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  14. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  15. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  16. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  17. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  18. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  19. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  20. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 75
  21. 2023Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokenscited by 71
  22. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 23
  23. 2025AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Modelscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models
  2. 2025CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs
  3. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  4. 2026QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for World Models and Video Generation
  5. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache
  6. 2026InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  8. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference