mlsanityData to 14 Oct 2026

Ensuring Fair LLM Serving Amid Diverse Applications

arXiv 2411.15997 ↗First posted 24 Nov 2024cs.LG cs.AI cs.DCcited by 3

Not on the map. Why?

Abstract (arXiv)

In a multi-tenant large language model (LLM) serving platform hosting diverse applications, some users may submit an excessive number of requests, causing the service to become unavailable to other users and creating unfairness. Existing fairness approaches do not account for variations in token lengths across applications and multiple LLM calls, making them unsuitable for such platforms. To address the fairness challenge, this paper analyzes millions of requests from thousands of users on MS CoPilot, a real-world multi-tenant LLM platform hosted by Microsoft. Our analysis confirms the inadequacy of existing methods and guides the development of FairServe, a system that ensures fair LLM access across diverse applications. FairServe proposes application-characteristic aware request throttling coupled with a weighted service counter based scheduling technique to curb abusive behavior and ensure fairness. Our experimental results on real-world traces demonstrate FairServe's superior performance compared to the state-of-the-art method in ensuring fairness. We are actively working on deploying our system in production, expecting to benefit millions of customers world-wide.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 24 Nov 2024.

Builds on

16 of 36 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2022Efficiently Scaling Transformer Inferencecited by 539
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  8. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  9. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  10. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  11. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  12. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
4 more
  1. 2023Fairness in Serving Large Language Modelscited by 78
  2. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  3. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  4. 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pamperingcited by 2
  2. 2025Efficient Serving of LLM Applications with Probabilistic Demand Modelingcited by 1
  3. 2026Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling
  2. 2025Equinox: Holistic Fair Scheduling in Serving Large Language Models
  3. 2025Locality-aware Fair Scheduling in LLM Serving
  4. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
  5. 2026WWW.Serve: Interconnecting Global LLM Services through Decentralization ↗
  6. 2026Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving
  7. 2024FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving
  8. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving