mlsanityData to 7 Oct 2026

Practical Online KV Cache Compaction for LLM Agents: An Empirical Study

arXiv 2608.00902 ↗First posted 2 Aug 2026cs.CLcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 2 more
Abstract (arXiv)

LLM agents accumulate long trajectories of reasoning steps, tool calls, and environment feedback, making the KV cache a major inference bottleneck. KV cache compaction can reduce this cost, but most prior methods assume a static context where future queries are known or can be approximated offline. Agents instead require online compaction: new information must be compressed before future relevance is known, using proxy queries cheap enough for the inference path. We study online compaction across token eviction (TE) and attention matching (AM), adapting both to compact agent turns and comparing cheap proxy sources such as boundary, repeat-prefill, and delayed future-generation queries. Experiments on BrowseComp-Plus and WideSearch show that immediate compaction often hurts performance, whereas delaying compaction to use the agent's future queries recovers much of the gap. Moreover, TE is often more robust than AM under imperfect proxies. Across models at different scales, TE preserves most of the accuracy while reducing KV cache by 80%, and can improve throughput over the no compaction baseline. These results position proxy-query selection as a core design choice for practical online KV compaction.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 26 of its 35 references that are in the map. First posted on arXiv 2 Aug 2026.

Builds on

26 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Think Clearly: Improving Reasoning via Redundant Token Pruning17 Jun

Other references · 25

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  10. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  11. 2023Adapting Language Models to Compress Contextscited by 350
  12. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
13 more
  1. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  2. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  3. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  5. 2024Long Context Compression with Activation Beaconcited by 101
  6. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  7. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  8. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  9. 2025WideSearch: Benchmarking Agentic Broad Info-Seekingcited by 66
  10. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 50
  11. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  12. 2026Fast KV Compaction via Attention Matchingcited by 28
  13. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Harness as a Language: A Minimalist Agent Framework With Maximal Expressivitycited by 0
  2. 2026Persistent Context Graphs for Efficient Memory Compaction in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026StepKV: Step-Aware KV Cache Compression for LLM Agents
  2. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
  3. 2026IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference
  4. 2026AgentKV: Phase-Aware KV Eviction for Agentic LLMs
  5. 2026TokenPilot: Cache-Efficient Context Management for LLM Agents
  6. 2026SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning
  7. 2026Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell
  8. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents