# WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

arXiv: https://arxiv.org/abs/2608.22704 · HTML: https://mlsanity.com/paper/2608.22704/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Placed in

- Efficient LLM inference and serving / [Audio and omni-modal token compression](https://mlsanity.com/inference/audio-and-omni-modal-token-compression/)

## Builds on (most cited first, 26 of 26)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://mlsanity.com/paper/2305.17118/) (2023)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference](https://mlsanity.com/paper/2407.11550/) (2024)
- [Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning](https://mlsanity.com/paper/2410.19258/) (2024)
- [ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression](https://mlsanity.com/paper/2412.03213/) (2024)
- [ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2502.00299/) (2025)
- [Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning](https://mlsanity.com/paper/2012.14768/) (2020)
- [SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs](https://mlsanity.com/paper/2506.05344/) (2025)
- [Long-Form Speech Generation with Spoken Language Models](https://mlsanity.com/paper/2412.18603/) (2024)
- [Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture](https://mlsanity.com/paper/2504.11809/) (2025)
- [LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation](https://mlsanity.com/paper/2603.10899/) (2026)
- [Assessing the Ability of Self-Attention Networks to Learn Word Order](https://mlsanity.com/paper/1906.00592/) (2019)
- [FASST: Fast LLM-based Simultaneous Speech Translation](https://mlsanity.com/paper/2408.09430/) (2024)
- [LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech](https://mlsanity.com/paper/2601.13539/) (2026)
- [Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models](https://mlsanity.com/paper/2604.06871/) (2026)
- [ExPosST: Explicit Positioning with Adaptive Masking for LLM-Based Simultaneous Machine Translation](https://mlsanity.com/paper/2603.14903/) (2026)
- [Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment](https://mlsanity.com/paper/2511.10670/) (2025)

## Abstract (arXiv)

Long-form audio inputs make the KV cache the dominant memory cost of speech LLMs. Prefill-only KV compression methods permanently discard audio KV positions once evicted, with no pathway to recover them during decoding. We show this is fragile on long-form audio: prefill attention concentrates near the audio start (an attention-sink effect), while decode-time attention distributes broadly, and the two rankings overlap weakly. We propose WnW (Waxing-and-Waning KV cache), which classifies KV-heads into anchor, tidal, and fixed roles via offline calibration. Anchor heads keep all audio KV on GPU and yield a decode-time signal of which audio region each token is read from; tidal heads keep a CPU-resident complement that is recalled chunk-by-chunk based on aggregated anchor-head scores; fixed heads keep only an on-GPU subset, with the rest permanently discarded. On LibriSpeech-Long with two 3B backbones (Voxtral-mini-3b and Qwen2.5-Omni-3B), WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate. Results generalize across language, task, and domain shifts, and CPU-GPU recall adds little decode-time overhead in our measurements.