mlsanityData to 14 Oct 2026

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

arXiv 2607.10582 ↗First posted 12 Jul 2026cs.LG cs.AIcited by 3

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck. Existing eviction policies generally apply the same attention- or recency-based rule to every token, ignoring semantic structure already available to the agent orchestrator. We introduce MemDecay, a training-free, region-aware KV-cache eviction policy. MemDecay assigns tokens region-specific base priorities and decay rates, refreshes retention scores when tokens receive attention, and evicts the lowest-scoring pages under a fixed cache budget while allowing critical regions to be pinned. We also provide a procedure for calibrating decay rates from measured attention lifetimes. We evaluate MemDecay at approximately 450 and 1,700 token contexts using Qwen2.5-1.5B and 3B. Across all settings, attention lifetimes differ by an order of magnitude across regions: system-token half-lives range from 148 to 189 decoding steps, compared with 14 to 16 for scratchpad tokens. Pinning preserves system-region facts at full-cache accuracy in every setting, while no baseline preserves more than 13 of 24. Region-aware retention remains effective as context grows, whereas recency-based retention collapses. Accumulated-attention retention performs better on unpinned content, however, and ablations identify attention-score normalization as the main limitation of the current formulation. These results establish semantic prompt structure as a robust signal for KV-cache management while clarifying how it should be combined with attention-based importance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 33 references that are in the map. First posted on arXiv 12 Jul 2026.

Builds on

28 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference6 Jun
  2. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov
  3. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul

Other references · 25

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  5. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  9. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  10. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  11. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  12. 2025From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMscited by 124
13 more
  1. 2024Transformers are Multi-State RNNscited by 120
  2. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  3. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  4. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74
  5. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  6. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  7. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43
  8. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  9. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 19
  10. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  11. 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Evictioncited by 4
  12. 2026Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Cachescited by 3
  13. 2026CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLMcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KVcited by 2
  2. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agentscited by 0
  3. 2026SparseEngine: Sparse-First Inference Enginecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TokenPilot: Cache-Efficient Context Management for LLM Agents
  2. 2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents
  3. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  4. 2026Learning to Evict from Key-Value Cache
  5. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems
  6. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  7. 2026StepKV: Step-Aware KV Cache Compression for LLM Agents
  8. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference