mlsanityData to 14 Oct 2026

FairBatching: Fairness-Aware Batch Formation for LLM Inference

arXiv 2510.14392 ↗First posted 16 Oct 2025cs.DC cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) inference systems face a fundamental tension between minimizing Time-to-First-Token (TTFT) latency for new requests and maintaining a high, steady token generation rate (low Time-Per-Output-Token, or TPOT) for ongoing requests. Existing stall-free batching schedulers proposed by Sarathi, while effective at preventing decode stalls, introduce significant computational unfairness. They prioritize decode tasks excessively, simultaneously leading to underutilized decode slack and unnecessary prefill queuing delays, which collectively degrade the system's overall quality of service (QoS). This work identifies the root cause of this unfairness: the non-monotonic nature of Time-Between-Tokens (TBT) as a scheduling metric and the rigid decode-prioritizing policy that fails to adapt to dynamic workload bursts. We therefore propose FairBatching, a novel LLM inference scheduler that enforces fair resource allocation between prefill and decode tasks. It features an adaptive batch capacity determination mechanism, which dynamically adjusts the computational budget to improve the GPU utilization without triggering SLO violations. Its fair and dynamic batch formation algorithm breaks away from the decode-prioritizing paradigm, allowing computation resources to be reclaimed from bursting decode tasks to serve prefill surges, achieving global fairness. Furthermore, FairBatching provides a novel load estimation method, enabling more effective coordination with upper-level schedulers. Implemented and evaluated on realistic traces, FairBatching significantly reduces TTFT tail latency by up to 2.29x while robustly maintaining TPOT SLOs, achieving overall 20.0% improvement in single-node capacity and 54.3% improvement in cluster-level capacity.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 19 of its 47 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

19 of 47 references.

References · 19

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  5. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  6. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  7. 2023Fairness in Serving Large Language Modelscited by 78
  8. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  9. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  10. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  11. 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 25
  12. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 24
7 more
  1. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  2. 2024Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloadscited by 17
  3. 2025Locality-aware Fair Scheduling in LLM Servingcited by 15
  4. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregationcited by 14
  5. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  6. 2025Hierarchical Autoscaling for Large Language Model Serving with Chironcited by 10
  7. 2025Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decodingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  2. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  3. 2026StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Servingcited by 0
  4. 2026Regulating Branch Parallelism in LLM Servingcited by 0
  5. 2026Human-Less LLM Serving: Quantifying the Human Tax on Throughputcited by 0
  6. 2026Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Servingcited by 0