mlsanityData to 14 Oct 2026

TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving

arXiv 2508.17219 ↗First posted 24 Aug 2025cs.DC cs.LGcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Prefix caching is crucial to accelerate multi-turn interactions and requests with shared prefixes. At the cluster level, existing prefix caching systems are tightly coupled with request scheduling to optimize cache efficiency and computation performance together, leading to load imbalance, data redundancy, and memory fragmentation of caching systems across instances. To address these issues, memory pooling is promising to shield the scheduler from the underlying cache management so that it can focus on the computation optimization. However, because existing prefix caching systems only transfer increasingly longer prefix caches between instances, they cannot achieve low-latency memory pooling. To address these problems, we propose a unified segment-level prefix cache pool, TokenLake. It uses a declarative cache interface to expose requests' query tensors, prefix caches, and cache-aware operations to TokenLake for efficient pooling. Powered by this abstraction, TokenLake can manage prefix cache at the segment level with a heavy-hitter-aware load balancing algorithm to achieve better cache load balance, deduplication, and defragmentation. TokenLake also transparently minimizes the communication volume of query tensors and new caches. Based on TokenLake, the scheduler can schedule requests elastically by using existing techniques without considering prefix cache management. Evaluations on real-world workloads show that TokenLake can improve throughput by up to 2.6× and 2.0× and boost hit rate by 2.0× and 2.1×, compared to state-of-the-art cache-aware routing and cache-centric PD-disaggregation solutions, respectively.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 61 references that are in the map. First posted on arXiv 24 Aug 2025.

Builds on

39 of 61 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding7 Jul
  2. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  4. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  6. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  7. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 31

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  7. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  8. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
  9. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  10. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  11. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  12. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
19 more
  1. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  2. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  3. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 54
  4. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  5. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  6. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 39
  7. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  8. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  9. 2025Serving Large Language Models on Huawei CloudMatrix384cited by 24
  10. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  11. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  12. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  13. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
  14. 2024Pie: Pooling CPU Memory for LLM Inferencecited by 13
  15. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  16. 2025InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceiverscited by 7
  17. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharingcited by 4
  18. 2024A System for Microserving of LLMscited by 3
  19. 2019DistCache: Provable Load Balancing for Large-Scale Storage Systems with Distributed Cachingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  2. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 14
  3. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 5
  4. 2025RepetitionCurse: Measuring and Understanding Router Imbalance in Mixture-of-Experts LLMs under DoS Stresscited by 2
  5. 2026NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
  2. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving
  3. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
  4. 2026PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
  5. 2026Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
  6. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
  7. 2026CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving
  8. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design