# OmniKVQuant: KV Cache Quantization for Omni-LLMs

arXiv: https://arxiv.org/abs/2609.11582 · HTML: https://mlsanity.com/paper/2609.11582/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 24 of its 26 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache quantization](https://mlsanity.com/inference/kv-cache-quantization/)

## Builds on (most cited first, 24 of 24)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs](https://mlsanity.com/paper/2502.04326/) (2025)
- [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://mlsanity.com/paper/2504.19874/) (2025)
- [Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities](https://mlsanity.com/paper/2505.17862/) (2025)
- [OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs](https://mlsanity.com/paper/2510.10689/) (2025)
- [video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models](https://mlsanity.com/paper/2506.15220/) (2025)
- [AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding](https://mlsanity.com/paper/2505.20862/) (2025)
- [OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models](https://mlsanity.com/paper/2511.14582/) (2025)
- [UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks](https://mlsanity.com/paper/2507.11336/) (2025)
- [DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Models](https://mlsanity.com/paper/2601.19267/) (2026)
- [On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs](https://mlsanity.com/paper/2603.14337/) (2026)
- [MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models](https://mlsanity.com/paper/2603.04800/) (2026)
- [OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization](https://mlsanity.com/paper/2605.21226/) (2026)
- [Out of Sight, Still in Mind: Token Compression for Omni-LLMs](https://mlsanity.com/paper/2607.21179/) (2026)

## Abstract (arXiv)

As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the de facto approach in text-only LLMs, but its application to Omni-LLMs remains unexplored. In this paper, we analyze how TurboQuant, a representative rotation-based KV cache quantization method, behaves on multimodal caches and identify two critical issues: temporal key drift and heterogeneous value geometry. To address these, we propose OmniKVQuant, a training-free framework that (i) sets the key quantization range over each short window of the input stream; and (ii) rotates values separately per modality. On Qwen2.5-Omni and Qwen3-Omni, OmniKVQuant enables 2-bit KV caches while substantially preserving performance across seven audio-visual benchmarks. We further provide a fused Triton decode kernel that unpacks the 2-bit cache during attention, so no dense FP16 cache is ever built. Code: https://github.com/kaistmm/OmniKVQuant