mlsanityData to 14 Oct 2026

CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents

arXiv 2606.16824 ↗First posted 15 Jun 2026cs.DC cs.OScited by 10

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

Coding agents are a fast-growing LLM application, executing as long-running closed-loop sessions in which LLM generations alternate with external tool calls. Yet, unlike chat workloads, their serving behavior has not been studied extensively. We address this gap by collecting a dataset of real-world coding assistant traces. Our analysis shows that coding agent sessions repeatedly reuse large prefixes and create sustained KVCache pressure that conventional LLM serving policies handle poorly. Based on our analysis, we present CacheWise, a KVCache management layer that improves KVCache reuse for coding agent workloads. CacheWise combines prefix-aware scheduling with reuse-aware eviction guided by lightweight predictions from tool call metadata. Implemented in vLLM and evaluated on the collected traces, CacheWise reduces KVCache evictions by up to 2-2.6x and improves total agent session completion time by up to 3.5x.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 42 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

26 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb

Other references · 25

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
13 more
  1. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  3. 2018Competitive caching with machine learned advicecited by 195
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  5. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  6. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 69
  7. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  8. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 25
  9. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  10. 2025T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planningcited by 14
  11. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 14
  12. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  13. 2025LLM Query Scheduling with Prefix Reuse and Latency Constraintscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Servingcited by 4
  2. 2026Crossflow: Prefill-Decode Elasticity for Agentic LLM Servingcited by 1
  3. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0
  4. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  5. 2026Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read Itcited by 0
  6. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systemscited by 0
  7. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reusecited by 0
  8. 2026AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelinescited by 0
  9. 2026Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learningcited by 0
  10. 2026Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Serving
  2. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
  3. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
  4. 2026Learning Agent Execution for KV-Cache Management in Agentic Serving
  5. 2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
  6. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
  7. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
  8. 2026TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving