# TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications

arXiv: https://arxiv.org/abs/2510.18586 · HTML: https://mlsanity.com/paper/2510.18586/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 38 of its 75 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- Efficient LLM inference and serving / [Efficient reasoning and agent inference](https://mlsanity.com/inference/efficient-reasoning-and-agent-inference/)

## Builds on (most cited first, 30 of 38)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://mlsanity.com/paper/2310.07240/) (2023)
- [TradingAgents: Multi-Agents LLM Financial Trading Framework](https://mlsanity.com/paper/2412.20138/) (2024)
- [AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society](https://mlsanity.com/paper/2502.08691/) (2025)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference](https://mlsanity.com/paper/2410.21465/) (2024)
- [AgentScope: A Flexible yet Robust Multi-Agent Platform](https://mlsanity.com/paper/2402.14034/) (2024)
- [Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference](https://mlsanity.com/paper/2308.12066/) (2023)
- [Parrot: Efficient Serving of LLM-based Applications with Semantic Variable](https://mlsanity.com/paper/2405.19888/) (2024)
- [Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://mlsanity.com/paper/2506.21506/) (2025)
- [Preble: Efficient Distributed Prompt Scheduling for LLM Serving](https://mlsanity.com/paper/2407.00023/) (2024)
- [Stateful Large Language Model Serving with Pensieve](https://mlsanity.com/paper/2312.05516/) (2023)
- [Autellix: An Efficient Serving Engine for LLM Agents as General Programs](https://mlsanity.com/paper/2502.13965/) (2025)
- [Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations](https://mlsanity.com/paper/2408.15232/) (2024)

## Built on by (most cited first, 16 of 16)

- [CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control](https://mlsanity.com/paper/2601.22705/) (2026)
- [TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing](https://mlsanity.com/paper/2604.03143/) (2026)
- [Libra: Efficient Resource Management for Agentic RL Post-Training](https://mlsanity.com/paper/2606.03077/) (2026)
- [ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference](https://mlsanity.com/paper/2605.27081/) (2026)
- [Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation](https://mlsanity.com/paper/2606.01725/) (2026)
- [Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving](https://mlsanity.com/paper/2606.02964/) (2026)
- [ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache](https://mlsanity.com/paper/2604.06370/) (2026)
- [A Policy-Driven Runtime Layer for Agentic LLM Serving](https://mlsanity.com/paper/2605.27744/) (2026)
- [Learning Agent Execution for KV-Cache Management in Agentic Serving](https://mlsanity.com/paper/2608.14624/) (2026)
- [Vulcan: Instance-specialized, Verifiable Systems Heuristics Through LLM-driven Search](https://mlsanity.com/paper/2512.25065/) (2025)
- [Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines](https://mlsanity.com/paper/2604.15186/) (2026)
- [Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scaling](https://mlsanity.com/paper/2604.17353/) (2026)
- [Scalable LLM Agent Tool Access in the Cloud](https://mlsanity.com/paper/2607.15593/) (2026)
- [BIO-MEMART: Biometric-Aware KV Cache Memory for Multi-User LLM Agents](https://mlsanity.com/paper/2609.08566/) (2026)
- [Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It](https://mlsanity.com/paper/2609.18849/) (2026)
- [Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving](https://mlsanity.com/paper/2609.25782/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed in complex multi-agent applications that rely on external function calls. This workload creates severe performance challenges for the KV Cache: spatial contention leads to the eviction of critical agents' caches and temporal underutilization leaves the cache of agents stalled on long-running function calls idling in GPU memory. We present TokenCake, a KV-Cache-centric serving framework that bridges this gap by co-optimizing scheduling and memory management through an agent-aware design. TokenCake's Temporal Scheduler employs an event-driven, opportunistic policy to proactively offload idle KV Caches during function calls and uses predictive uploading to hide data transfer latency. TokenCake's Spatial Scheduler uses dynamic memory partitioning, guided by a hybrid priority metric combining graph structure and runtime state, to reserve GPU memory for critical-path agents. Our evaluation on representative multi-agent benchmarks shows that TokenCake reduces end-to-end latency by over 47.06% and improves effective GPU memory utilization by up to 16.9% compared to vLLM.