Data to 7 Oct 2026

Fairness in Serving Large Language Models

arXiv 2401.00588 ↗First posted 31 Dec 2023cs.AI cs.LG cs.PFcited by 77

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

High-demand LLM inference services (e.g., ChatGPT and BARD) support a wide range of requests from short chat conversations to long document reading. To ensure that all client requests are processed fairly, most major LLM inference services have request rate limits, to ensure that no client can dominate the request queue. However, this rudimentary notion of fairness also results in under-utilization of the resources and poor client experience when there is spare capacity. While there is a rich literature on fair scheduling, serving LLMs presents new challenges due to their unpredictable request lengths and their unique batching characteristics on parallel accelerators. This paper introduces the definition of LLM serving fairness based on a cost function that accounts for the number of input and output tokens processed. To achieve fairness in serving, we propose a novel scheduling algorithm, the Virtual Token Counter (VTC), a fair scheduler based on the continuous batching mechanism. We prove a 2x tight upper bound on the service difference between two backlogged clients, adhering to the requirement of work-conserving. Through extensive experiments, we demonstrate the superior performance of VTC in ensuring fairness, especially in contrast to other baseline methods, which exhibit shortcomings under various conditions. The reproducible code is available at https://github.com/Ying1123/VTC-artifact

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 51 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

15 of 51 references.

Closest ancestors · 1

Same direction, newest first

2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 14

2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2022Efficiently Scaling Transformer Inferencecited by 529
2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68

2 more

2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 23; 2026: 26.

77 papers.

Sort:

By area:

2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
2024A Survey on Efficient Inference for Large Language Modelscited by 197
2024Efficient LLM Scheduling by Learning to Rankcited by 83
2024Safeguarding Large Language Models: A Surveycited by 73
2024Building Guardrails for Large Language Modelscited by 61
2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 49
2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31

38 more

2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
2025ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinkingcited by 22
2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 15
2024AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Executioncited by 14
2024Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inferencecited by 13
2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 12
2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inferencecited by 6
2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6
2024MPC-Minimized Secure LLM Inferencecited by 5
2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Servingcited by 5
2024FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Servingcited by 5
2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 5
2024Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMscited by 4
2025Ascendra: Dynamic Request Prioritization for Efficient LLM Servingcited by 4
2024Ensuring Fair LLM Serving Amid Diverse Applicationscited by 3
2025What Limits Agentic Systems Efficiency?cited by 3
2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
2025Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimizationcited by 2
2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applicationscited by 2
2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systemscited by 2
2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
2026Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Modelcited by 2
2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
2026Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Servingcited by 2
2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way