mlsanityData to 7 Oct 2026

EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?

arXiv 2609.33762 ↗First posted 27 Sep 2026cs.DC cs.LGcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM agents resend their whole conversation on every turn, and most of it was already processed on the previous turn. Serving systems avoid recomputing it by caching its key-value (KV) state and, when GPU memory runs out, by offloading that state to host memory. For agents, offloading gives inconsistent results: on the same coding-agent workload it speeds up one deployment, slows down another, and changes nothing on a third, even where loading a token back is several times cheaper than recomputing it. The reason is that cached state must survive until it is used again. While one agent waits for its tool, the server processes the contexts of all other agents, so an agent's prefix is reused only if the host tier holds the reusable context of the whole agent pool, which we call the reuse working set. A smaller tier keeps writing state that is evicted before anyone reads it. We present EfficientAgent, which sizes and manages the host tier by this working set. A stack-distance model estimates the working set from agent histories to size the host tier; its predictions, made before the experiments, located the capacity at which offloading starts to pay. When the tier is too small, a runtime policy stops writing large refills of evicted context and keeps extending prefixes that are still cached; when the tier is large enough, it writes everything. On SWE-bench Verified coding agents, a host tier sized to the estimated working set cuts recomputed prompt tokens by 93% and end-to-end time by 39%. With a small fixed tier, the policy cuts recomputation by 35%; with a large tier, it avoids the 4.3-fold increase caused by always filtering writes. Across three GPU types and two models, offloading pays off when the GPU has little compute per byte of host bandwidth and the host tier holds the working set. Code is available at https://github.com/KunmingSHAO/efficientagent_release.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 39 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

24 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving26 Aug
  2. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents23 Jan
  3. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov
  4. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul
  5. 2024Training Software Engineering Agents and Verifiers with SWE-Gym30 Dec
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 17

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  8. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  9. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 136
  10. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  11. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  12. 2023Stateful Large Language Model Serving with Pensievecited by 59
5 more
  1. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 44
  2. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  3. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 32
  4. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  5. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
  2. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents
  3. 2026Workload-Aware Caching for Multi-Agent Systems
  4. 2026Learning Agent Execution for KV-Cache Management in Agentic Serving
  5. 2026Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management
  6. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents
  7. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
  8. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control