mlsanityData to 14 Oct 2026

OmniKVQuant: KV Cache Quantization for Omni-LLMs

arXiv 2609.11582 ↗First posted 10 Sep 2026cs.CVcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the de facto approach in text-only LLMs, but its application to Omni-LLMs remains unexplored. In this paper, we analyze how TurboQuant, a representative rotation-based KV cache quantization method, behaves on multimodal caches and identify two critical issues: temporal key drift and heterogeneous value geometry. To address these, we propose OmniKVQuant, a training-free framework that (i) sets the key quantization range over each short window of the input stream; and (ii) rotates values separately per modality. On Qwen2.5-Omni and Qwen3-Omni, OmniKVQuant enables 2-bit KV caches while substantially preserving performance across seven audio-visual benchmarks. We further provide a fused Triton decode kernel that unpacks the 2-bit cache during attention, so no dense FP16 cache is ever built. Code: https://github.com/kaistmm/OmniKVQuant

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 26 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

24 of 26 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization20 May
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 21

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2025Qwen2.5-Omni Technical Reportcited by 736
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  9. 2025Qwen3-Omni Technical Reportcited by 497
  10. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  11. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  12. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
9 more
  1. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  2. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 47
  3. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 30
  4. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 22
  5. 2025UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarkscited by 19
  6. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Modelscited by 13
  7. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6
  8. 2026MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelscited by 5
  9. 2026Out of Sight, Still in Mind: Token Compression for Omni-LLMscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs
  2. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  3. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
  4. 2025XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
  5. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  6. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
  7. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models
  8. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction