mlsanityData to 7 Oct 2026

Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling

arXiv 2603.15202 ↗First posted 16 Mar 2026cs.DC cs.OScited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

High-quality LLM request scheduling requires meeting two key objectives: ensuring the routed instance has KVCache to accelerate request execution, and ensuring that the workload is balanced across instances. Achieving both objectives is challenging because pursuing one may compromise the other. Current approaches use various combinators (e.g., linear combinations) to compute a scheduling score that combines indicators for the two objectives. These approaches are complex: they either require significant workload-specific hyperparameter tuning or model-hardware-aware simulator development, yet could still lead to suboptimal performance. In this paper, we show that using a simple multiplication of two carefully chosen indicators: one KVCache-aware (new prefill tokens if routed to an instance) and one load-balancing-aware (current batch size of the instance), as the scheduling score (LMETRIC) can achieve both objectives simultaneously without any hyperparameter tuning. The key idea is that the simply multiplied score considers both objectives in a manner similar to a linear combination, but the original hyperparameters cancel out during comparison, so no tuning is needed to find the best parameters. The two indicators are chosen based on our analysis of LLM characteristics. Our extensive experiments show that this simple approach can reduce TTFT by 92% and 39%, and TPOT by 24% and 51%, compared to vLLM-v1 and an in-production scheduler on real-world workloads covering chatbots and coding agents. We also derive the mathematical conditions under which multiplication may fail, and find that such conditions are extremely rare in practice and can be detected (and mitigated) beforehand. LMETRIC has been deployed in production and canary release confirms its effectiveness

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 49 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

23 of 49 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving6 Feb
  2. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity3 Dec
  3. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications14 Nov
  4. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
  5. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  7. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  9. 2023Fairness in Serving Large Language Models31 Dec
  10. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 13

  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  3. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  4. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  5. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  6. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  7. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  8. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  9. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  10. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  11. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  12. 2025PolyServe: Efficient Multi-SLO Serving at Scalecited by 5
1 more
  1. 2025BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architecturescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  2. 2026TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  2. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving
  3. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving
  4. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
  5. 2026LAPS: A Length-Aware-Prefill LLM Serving System
  6. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees
  7. 2026Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling
  8. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Serving