mlsanityData to 14 Oct 2026

HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models

arXiv 2609.02029 ↗First posted 2 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Long-context inference retains a growing key--value (KV) cache during decoding, which consumes substantial GPU memory and can reduce generation throughput. This bottleneck remains in hybrid language models because their residual global-attention layers can dominate context-dependent cache demand. We study how to allocate this state under an aggregate KV-residency budget. We introduce HeadWiseKV, a training-free framework that compresses the residual global KV caches of hybrid language models while preserving their native local, recurrent, and linear paths. It assigns each physical KV head a static, multilevel history window, making cache demand predictable before serving. We formulate this allocation as a restricted operational rate--distortion problem and propose SeqCalib as the core policy-generation algorithm in HeadWiseKV. SeqCalib processes layers in execution order and conditions each decision on the lower-layer policy used at deployment, thereby accounting for interactions across depth. A grouped-cache runtime materializes the selected policy as actual per-head KV residency rather than a mask over a full cache. We evaluate downstream quality across four hybrid long-context models and study physical residency and serving behavior on Qwen3.6-27B. HeadWiseKV retains near-Full-KV RULER and LoCoMo quality across the evaluated models. In the fixed-model systems study, it reduces sampled peak device memory by 8.59% at a 112K context length and extends the largest verified successful context from 114K to 161K.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

24 of 31 references.

References · 24

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  7. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  8. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  11. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
12 more
  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  2. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  4. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  5. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 100
  6. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  7. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 97
  8. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 58
  9. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
  10. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  11. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 23
  12. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Headcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  2. 2026HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
  3. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  4. 2026CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
  5. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  6. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  7. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
  8. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading