mlsanityData to 7 Oct 2026

TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications

arXiv 2510.18586 ↗First posted 21 Oct 2025cs.DCcited by 16

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed in complex multi-agent applications that rely on external function calls. This workload creates severe performance challenges for the KV Cache: spatial contention leads to the eviction of critical agents' caches and temporal underutilization leaves the cache of agents stalled on long-running function calls idling in GPU memory. We present TokenCake, a KV-Cache-centric serving framework that bridges this gap by co-optimizing scheduling and memory management through an agent-aware design. TokenCake's Temporal Scheduler employs an event-driven, opportunistic policy to proactively offload idle KV Caches during function calls and uses predictive uploading to hide data transfer latency. TokenCake's Spatial Scheduler uses dynamic memory partitioning, guided by a hybrid priority metric combining graph structure and runtime state, to reserve GPU memory for critical-path agents. Our evaluation on representative multi-agent benchmarks shows that TokenCake reduces end-to-end latency by over 47.06% and improves effective GPU memory utilization by up to 16.9% compared to vLLM.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 75 references that are in the map. First posted on arXiv 21 Oct 2025.

Builds on

38 of 75 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb

Other references · 37

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  9. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  10. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  11. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  12. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
25 more
  1. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  2. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  4. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  6. 2024TradingAgents: Multi-Agents LLM Financial Trading Frameworkcited by 182
  7. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 180
  8. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  9. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
  10. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  11. 2024AgentScope: A Flexible yet Robust Multi-Agent Platformcited by 120
  12. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  13. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  14. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 75
  15. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  16. 2023Stateful Large Language Model Serving with Pensievecited by 59
  17. 2024Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversationscited by 43
  18. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  19. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  20. 2025MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Usecited by 37
  21. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  22. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  23. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  24. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  25. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 15.

16 papers.

Sort:

By area:

  1. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  2. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharingcited by 8
  3. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  4. 2026A Policy-Driven Runtime Layer for Agentic LLM Servingcited by 3
  5. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  6. 2026ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cachecited by 2
  7. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 2
  8. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 2
  9. 2025Vulcan: Instance-specialized, Verifiable Systems Heuristics Through LLM-driven Searchcited by 1
  10. 2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelinescited by 1
  11. 2026Learning Agent Execution for KV-Cache Management in Agentic Servingcited by 1
  12. 2026Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scalingcited by 0
4 more
  1. 2026Scalable LLM Agent Tool Access in the Cloudcited by 0
  2. 2026BIO-MEMART: Biometric-Aware KV Cache Memory for Multi-User LLM Agentscited by 0
  3. 2026Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read Itcited by 0
  4. 2026Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
  2. 2026Pancake: Hierarchical Memory System for Multi-Agent LLM Serving
  3. 2026MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems
  4. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management
  5. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
  6. 2026TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving
  7. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents
  8. 2026KVCMAS: Efficient KV cache Correction for Shared Context in Multi-Agent Systems