# FairBatching: Fairness-Aware Batch Formation for LLM Inference

arXiv: https://arxiv.org/abs/2510.14392 · HTML: https://mlsanity.com/paper/2510.14392/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Builds on (most cited first, 19 of 19)

- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://mlsanity.com/paper/2408.12757/) (2024)
- [Fairness in Serving Large Language Models](https://mlsanity.com/paper/2401.00588/) (2023)
- [MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool](https://mlsanity.com/paper/2406.17565/) (2024)
- [ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference](https://mlsanity.com/paper/2404.07947/) (2024)
- [KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider](https://mlsanity.com/paper/2506.02634/) (2025)
- [BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Caching](https://mlsanity.com/paper/2412.17246/) (2024)
- [Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning](https://mlsanity.com/paper/2505.04021/) (2025)
- [Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving](https://mlsanity.com/paper/2508.01989/) (2025)
- [Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads](https://mlsanity.com/paper/2407.00010/) (2024)
- [Locality-aware Fair Scheduling in LLM Serving](https://mlsanity.com/paper/2501.14312/) (2025)
- [Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation](https://mlsanity.com/paper/2503.20552/) (2025)
- [Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation](https://mlsanity.com/paper/2405.01814/) (2024)
- [Hierarchical Autoscaling for Large Language Model Serving with Chiron](https://mlsanity.com/paper/2501.08090/) (2025)
- [Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding](https://mlsanity.com/paper/2502.05609/) (2025)

## Built on by (most cited first, 6 of 6)

- [Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling](https://mlsanity.com/paper/2603.15202/) (2026)
- [ProServe: Unified Multi-Priority Request Scheduling for LLM Serving](https://mlsanity.com/paper/2512.12928/) (2025)
- [StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving](https://mlsanity.com/paper/2604.09562/) (2026)
- [Regulating Branch Parallelism in LLM Serving](https://mlsanity.com/paper/2605.06914/) (2026)
- [Human-Less LLM Serving: Quantifying the Human Tax on Throughput](https://mlsanity.com/paper/2606.20577/) (2026)
- [Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving](https://mlsanity.com/paper/2609.18112/) (2026)

## Abstract (arXiv)

Large language model (LLM) inference systems face a fundamental tension between minimizing Time-to-First-Token (TTFT) latency for new requests and maintaining a high, steady token generation rate (low Time-Per-Output-Token, or TPOT) for ongoing requests. Existing stall-free batching schedulers proposed by Sarathi, while effective at preventing decode stalls, introduce significant computational unfairness. They prioritize decode tasks excessively, simultaneously leading to underutilized decode slack and unnecessary prefill queuing delays, which collectively degrade the system's overall quality of service (QoS). This work identifies the root cause of this unfairness: the non-monotonic nature of Time-Between-Tokens (TBT) as a scheduling metric and the rigid decode-prioritizing policy that fails to adapt to dynamic workload bursts. We therefore propose FairBatching, a novel LLM inference scheduler that enforces fair resource allocation between prefill and decode tasks. It features an adaptive batch capacity determination mechanism, which dynamically adjusts the computational budget to improve the GPU utilization without triggering SLO violations. Its fair and dynamic batch formation algorithm breaks away from the decode-prioritizing paradigm, allowing computation resources to be reclaimed from bursting decode tasks to serve prefill surges, achieving global fairness. Furthermore, FairBatching provides a novel load estimation method, enabling more effective coordination with upper-level schedulers. Implemented and evaluated on realistic traces, FairBatching significantly reduces TTFT tail latency by up to 2.29x while robustly maintaining TPOT SLOs, achieving overall 20.0% improvement in single-node capacity and 54.3% improvement in cluster-level capacity.