mlsanityData to 7 Oct 2026

DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving

arXiv 2602.06502 ↗First posted 6 Feb 2026cs.DCcited by 8

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

In LLM serving, reusing the KV cache of prompts across requests is critical for reducing TTFT and serving costs. Cache-affinity scheduling, which co-locates requests with the same prompt prefix to maximize KV cache reuse, often conflicts with load-balancing scheduling that distributes requests evenly across compute instances. Existing schedulers fail to reconcile this trade-off as they operate within a single mapping space, typically applying cache-affinity routing to a subset of requests and load-balanced routing to the rest, without a unified solution to achieve both goals. To address this limitation, we propose DualMap, a dual-mapping scheduling strategy for distributed LLM serving that achieves both cache affinity and load balancing. Its key idea is to map each request to two candidate instances via two independent hash functions based on the request prompt, then intelligently select the better candidate based on current system states. This design increases the likelihood that requests with shared prefixes are co-located, while evenly dispersing distinct prefixes across the cluster via ``the power of two choices''. To make DualMap robust under dynamic and skewed real-world workloads, we incorporate three techniques: 1) SLO-aware request routing, which prioritizes cache affinity but switches to load-aware scheduling when TTFT exceeds the SLO, enhancing load balance without sacrificing cache reuse; 2) hotspot-aware rebalancing, which dynamically migrates requests from overloaded to underloaded instances, mitigating hotspots and rebalancing the system; 3) lightweight dual-hash-ring scaling, which leverages a dual-hash-ring mapping to support fast and low-overhead instance scaling without costly global remapping. Experiments on real-world workloads show that DualMap improves effective request capacity by up to 2.25× under the same TTFT SLO constraints compared with SOTA work.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 39 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

28 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Locality-aware Fair Scheduling in LLM Serving24 Jan
  2. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  3. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  4. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 23

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  7. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  8. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  10. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  11. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  12. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
11 more
  1. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  2. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  3. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  4. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 52
  5. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  6. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43
  7. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  8. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  9. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
  10. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 21
  11. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inferencecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  2. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  3. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  4. 2026CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Servingcited by 1
  5. 2026Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Servingcited by 0
  6. 2026Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMscited by 0
  7. 2026TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructurecited by 0
  8. 2026Affix Cache for Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
  2. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  3. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  4. 2026PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
  5. 2026Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale
  6. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  7. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving
  8. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration