mlsanityData to 14 Oct 2026

CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving

arXiv 2608.19677 ↗First posted 20 Aug 2026cs.DC cs.LGcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Prefix caching avoids prefill only when a repeated request returns to a server that still holds the prefix KV. Cache-blind balancing disperses that reuse; fixed affinity preserves it but can overload a server. CacheRoute resolves this tradeoff with a periodic routing plan. It admits high-rate keys to a stable warm set and places their assignments by expected load. Hot keys may use more than one destination, although every key in our primary semi-synthetic aggregate uses exactly one. On Llama-3.3-70B in fp8 across 60 H100 GPUs, CacheRoute sustains 176+/-11 QPS at a 3.5-s p99 SLO, 2.3x the strongest of five baselines. Served KV-cache hit rate rises from 64.1+/-1.3% under cache-blind balancing to 93.2+/-0.5%. A second semi-synthetic aggregate and controlled 8B and burst experiments separate the effects of affinity and placement. Two 32B workloads provide the counterexamples: when affinity recovers too little KV work, its residual load skew reduces or erases the improvement. We therefore recommend gating any deployment with a shadow replay rather than enabling affinity from workload statistics alone.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 9 of its 14 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

9 of 14 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 8

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  6. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  7. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  8. 2016Consistent Hashing with Bounded Loadscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving
  2. 2026Characterizing Contention-Induced Reliability Collapse in KV-Cache Timing Side Channels for Multi-Tenant LLM Serving
  3. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
  4. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
  5. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  6. 2026PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving
  7. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
  8. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration