mlsanityData to 14 Oct 2026

TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing

arXiv 2604.03143 ↗First posted 3 Apr 2026cs.DCcited by 8

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

Multi-agent LLM applications organize execution in synchronized rounds where a central scheduler gathers outputs from all agents and redistributes the combined context. This All-Gather communication pattern creates massive KV Cache redundancy, because every agent's prompt contains the same shared output blocks, yet existing reuse methods fail to exploit it efficiently. We present TokenDance, a system that scales the number of concurrent agents by exploiting the All-Gather pattern for collective KV Cache sharing. TokenDance's KV Collector performs KV Cache reuse over the full round in one collective step, so the cost of reusing a shared block is paid once regardless of agent count. Its Diff-Aware Storage encodes sibling caches as block-sparse diffs against a single master copy, achieving 11-17x compression on representative workloads. Evaluation on GenerativeAgents and AgentSociety shows that TokenDance supports up to 2.7x more concurrent agents than vLLM with prefix caching under SLO requirement, reduces per-agent KV Cache storage by up to 17.5x, and achieves up to 1.9x prefill speedup over per-request position-independent caching.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 40 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

25 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications21 Oct
  2. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb
  3. 2024DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving5 Nov

Other references · 22

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  11. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  12. 2018VirtualHome: Simulating Household Activities via Programscited by 415
10 more
  1. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  4. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  7. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  8. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  9. 2026What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproductioncited by 6
  10. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Beyond tokens: a unified framework for latent communication in LLM-based multi-agent systemscited by 7
  2. 2026Token Economics for LLM Agents: A Dual-View Study from Computing and Economicscited by 6
  3. 2026LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systemscited by 6
  4. 2026Architectural Implications of Agentic AI Workflowscited by 2
  5. 2026BIO-MEMART: Biometric-Aware KV Cache Memory for Multi-User LLM Agentscited by 0
  6. 2026PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstructioncited by 0
  7. 2026KVCMAS: Efficient KV cache Correction for Shared Context in Multi-Agent Systemscited by 0
  8. 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
  2. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents
  3. 2026Learning Agent Execution for KV-Cache Management in Agentic Serving
  4. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents
  5. 2026Pancake: Hierarchical Memory System for Multi-Agent LLM Serving
  6. 2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse
  7. 2026A Policy-Driven Runtime Layer for Agentic LLM Serving
  8. 2026[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows