mlsanityData to 14 Oct 2026

AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systems

arXiv 2607.21604 ↗First posted 15 May 2026cs.AIcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

Memory-augmented LLM agents maintain context across hundreds of interactions through agentic memory systems that actively curate retrieved content with LLM-generated metadata such as summaries, keywords, and tags. From an inference cost standpoint, every retrieval triggers a full re-encoding of these structured memory units into Key-Value (KV) states, which dominates prefill latency. Existing training-free KV reuse methods mitigate this by selectively recomputing a small fraction of tokens, but were designed for RAG-style raw passages and degrade on structured agentic memories. We present AgentKVShift, a training-free, probe-guided KV residual correction method that operates per retrieved memory unit. One of the crucial insights we demonstrate is that the per-memory KV reuse residual decomposes into a shared memory-level offset plus small token-wise fluctuations. Estimating this offset from a small probe set allows us to correct every reused token by a single weighted correction. Unlike prior reuse methods which decide which tokens to recompute and leave the rest of the cache stale, AgentKVShift also corrects the tokens it does not recompute, turning the refresh budget into useful signal across the entire chunk. Across four open-source LLMs spanning 3B to 32B parameters and two long-horizon agentic memory benchmarks (long-term dialogue and agentic applications), AgentKVShift achieves near full recompute performance while refreshing only 10-30% of the cache, outperforming baselines at the same recompute ratio. It requires up to 5x lower recompute to reach this near-full performance, which prior reuse methods only attain at 45-55% refresh. In this regime, AgentKVShift delivers prefill speedups of 2-3.5x over no-KV-reuse on a single A100. AgentKVShift orthogonally composes with KV cache quantization, retaining over 2x the F1 of prior reuse methods under aggressive 2- and 4-bit settings.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 28 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

24 of 28 references.

References · 24

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  7. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  8. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  12. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memorycited by 400
12 more
  1. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  2. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  3. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 237
  4. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  5. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 146
  6. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 122
  7. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 53
  8. 2025LiCoMemory: Lightweight and Cognitive Agentic Memory for Efficient Long-Term Reasoningcited by 38
  9. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
  10. 2025Softmax is 1/2-Lipschitz: A tight bound across all ℓp normscited by 22
  11. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 18
  12. 2026ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  2. 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
  2. 2026LatentRemedy: Efficient Remedy For Non-Prefix KV Cache Reuse in Multi-Agent Systems
  3. 2026KVCMAS: Efficient KV cache Correction for Shared Context in Multi-Agent Systems
  4. 2026TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
  5. 2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents
  6. 2026AgentKV: Phase-Aware KV Eviction for Agentic LLMs
  7. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems
  8. 2026PatchKV: Efficient KV Cache Recovery for Dynamically Edited LLM Contexts