mlsanityData to 14 Oct 2026

Locality-aware Fair Scheduling in LLM Serving

arXiv 2501.14312 ↗First posted 24 Jan 2025cs.DC cs.LGcited by 15

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model (LLM) inference workload dominates a wide variety of modern AI applications, ranging from multi-turn conversation to document analysis. Balancing fairness and efficiency is critical for managing diverse client workloads with varying prefix patterns. Unfortunately, existing fair scheduling algorithms for LLM serving, such as Virtual Token Counter (VTC), fail to take prefix locality into consideration and thus suffer from poor performance. On the other hand, locality-aware scheduling algorithms in existing LLM serving frameworks tend to maximize the prefix cache hit rate without considering fair sharing among clients. This paper introduces the first locality-aware fair scheduling algorithm, Deficit Longest Prefix Match (DLPM), which can maintain a high degree of prefix locality with a fairness guarantee. We also introduce a novel algorithm, Double Deficit LPM (D2LPM), extending DLPM for the distributed setup that can find a balance point among fairness, locality, and load-balancing. Our extensive evaluation demonstrates the superior performance of DLPM and D2LPM in ensuring fairness while maintaining high throughput (up to 2.87× higher than VTC) and low per-client (up to 7.18× lower than state-of-the-art distributed LLM serving system) latency.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. First posted on arXiv 24 Jan 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  2. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  3. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  4. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  5. 2025Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inferencecited by 2
  6. 2025CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancingcited by 1
  7. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 1
  8. 2026Scheduling Mixed RL Rollouts Beyond Prefix Localitycited by 1
  9. 2025Alchemist: Towards the Design of Efficient Online Continual Learning Systemcited by 0
  10. 2026FMplex: Model Virtualization for Serving Extensible Foundation Modelscited by 0
  11. 2026Recency/Frequency Adaptive KV Caching for Large Language Model Servingcited by 0
  12. 2026PEEK: Predictive Queue-Informed KV Cache Management for LLM Servingcited by 0
3 more
  1. 2026Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Servingcited by 0
  2. 2026Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read Itcited by 0
  3. 2026Inference Auctionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Fairness in Serving Large Language Models
  2. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  3. 2026Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling
  4. 2025SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference
  5. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  6. 2026Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice
  7. 2026Multi-Layer Scheduling for MoE-Based LLM Reasoning
  8. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving