mlsanityData to 7 Oct 2026

WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

arXiv 2608.22704 ↗First posted 24 Aug 2026cs.CL cs.SDcited by 0

On the map: Efficient LLM inference and serving → Audio and omni-modal token compression

Abstract (arXiv)

Long-form audio inputs make the KV cache the dominant memory cost of speech LLMs. Prefill-only KV compression methods permanently discard audio KV positions once evicted, with no pathway to recover them during decoding. We show this is fragile on long-form audio: prefill attention concentrates near the audio start (an attention-sink effect), while decode-time attention distributes broadly, and the two rankings overlap weakly. We propose WnW (Waxing-and-Waning KV cache), which classifies KV-heads into anchor, tidal, and fixed roles via offline calibration. Anchor heads keep all audio KV on GPU and yield a decode-time signal of which audio region each token is read from; tidal heads keep a CPU-resident complement that is recalled chunk-by-chunk based on aggregated anchor-head scores; fixed heads keep only an on-GPU subset, with the rest permanently discarded. On LibriSpeech-Long with two 3B backbones (Voxtral-mini-3b and Qwen2.5-Omni-3B), WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate. Results generalize across language, task, and domain shifts, and CPU-GPU recall adds little decode-time overhead in our measurements.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 37 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

26 of 37 references.

References · 26

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  10. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  12. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
14 more
  1. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  2. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  3. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  4. 2020Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learningcited by 26
  5. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMscited by 21
  6. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  7. 2025Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecturecited by 12
  8. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 11
  9. 2019Assessing the Ability of Self-Attention Networks to Learn Word Ordercited by 9
  10. 2024FASST: Fast LLM-based Simultaneous Speech Translationcited by 7
  11. 2026LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speechcited by 5
  12. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Modelscited by 4
  13. 2026ExPosST: Explicit Positioning with Adaptive Masking for LLM-Based Simultaneous Machine Translationcited by 2
  14. 2025Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignmentcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference
  2. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
  3. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
  4. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
  5. 2026HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
  6. 2026Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper
  7. 2026Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs
  8. 2026Acoustic-to-Text KV Compression for Full-Duplex Speech Models