mlsanityData to 7 Oct 2026

PTStore (Prefix Tensor Store): Distributed Prefix Caching and Replication for High Throughput Inference Serving

arXiv 2607.22648 ↗First posted 25 Jun 2026cs.AI cs.DCcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Inspired by the design of client caching in Content Delivery Networks (CDNs), PTStore distributes and replicates popular tensors that form reusable KV cache prefixes, which are the main technique used by state of art approaches to accelerate inferences. This reduces the latency of accessing the KV cache and alleviates load imbalance caused by a disproportionately large number of requests on servers containing popular tensors. Furthermore, thanks to decentralization, PTStore allows the expansion of the size of the KV cache for LLM inference by orders of magnitude. As a result, PTStore can execute inferences on long passage Q&A datasets 5-6 times more efficiently than current baselines, which do not aggregate memory across different nodes and GPUs and therefore require regenerating the KV cache.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 26 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

11 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 9

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2020Knowledge Distillation: A Surveycited by 1,578
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2022Efficiently Scaling Transformer Inferencecited by 529
  8. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  9. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reusecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
  2. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
  3. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
  4. 2024Compute Or Load KV Cache? Why Not Both?
  5. 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
  6. 2026The KV Cache Is the New Memory Wall
  7. 2025Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference
  8. 2026Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference