mlsanityData to 14 Oct 2026

Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems

arXiv 2608.18098 ↗First posted 9 Jun 2026cs.CL cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Key-value (KV) caching is essential for efficient autoregressive inference in transformer based dialog systems, yet existing strategies treat all cached entries uniformly or apply coarse eviction heuristics that fail to adapt as dialog topics evolve. We propose Fractional Decay KV-Cache (FD-KVC), a novel algorithm that maintains a dual-channel scoring mechanism for each cached KV pair: a cumulative attention channel that tracks aggregate importance (akin to H2O), and a recency-weighted relevance channel governed by temporal decay and reinforcement-inspired updates. The combination enables FD-KVC to both preserve historically important tokens and rapidly adapt when dialog topics shift. An adaptive learning rate driven by an ownership loss function ensures convergence without oscillation. FD-KVC operates entirely on CPU with negligible overhead. Across five diverse multi-turn dialog scenarios with 600 dialogs each, FD-KVC outperforms H2O, the state-of-the-art heavy-hitter baseline, by +6.7% on composite late-turn alignment, with improvements of +127% on topic-shift, +87% on gradual evolution, and +30% on mixed-topic dialogs. FD-KVC adapts to new topics 3.6X faster than H2O and achieves the highest topic diversity (80.6%) across all methods. Ablation studies confirm the contribution of each component.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 21 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

12 of 21 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 8

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  6. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  7. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  8. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs
  2. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  3. 2026Learning to Evict from Key-Value Cache
  4. 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management
  5. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
  6. 2025Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing
  7. 2025EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments
  8. 2026StepKV: Step-Aware KV Cache Compression for LLM Agents