mlsanityData to 14 Oct 2026

Equinox: Holistic Fair Scheduling in Serving Large Language Models

arXiv 2508.16646 ↗First posted 19 Aug 2025cs.DC cs.AIcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

We address the limitations of current LLM serving with a dual-counter framework separating user and operator perspectives. The User Fairness Counter measures quality of service via weighted tokens and latency; the Resource Fairness Counter measures operational efficiency through throughput and GPU utilization. Since these metrics are only available post-execution, creating a scheduling paradox, we introduce a deterministic Mixture of Prediction Experts (MoPE) framework to predict user-perceived latency, output tokens, throughput, and GPU utilization. These predictions enable calculation of a unified Holistic Fairness score that balances both counters through tunable parameters for proactive fairness-aware scheduling. We implement this in Equinox, an open-source system with other optimizations like adaptive batching, and stall-free scheduling. Evaluations on production traces (ShareGPT, LMSYS) and synthetic workloads demonstrate Equinox achieves up to 1.3× higher throughput, 60% lower time-to-first-token latency, and 13% higher fairness versus VTC while maintaining 94% GPU utilization, proving fairness under bounded discrepancy across heterogeneous platforms.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 46 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

22 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  2. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  8. 2022Predictability and Surprise in Large Generative Modelscited by 210
  9. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 137
  10. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  11. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  12. 2024LLM Processes: Numerical Predictive Distributions Conditioned on Natural Languagecited by 55
6 more
  1. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 33
  2. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20
  3. 2020Hierarchical Roofline Performance Analysis for Deep Learning Applicationscited by 11
  4. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  5. 2024KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generationcited by 6
  6. 2025Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
  2. 2025AdaFair-MARL: Enforcing Adaptive Fairness Constraints in Multi-Agent Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling
  2. 2026Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving
  3. 2023Fairness in Serving Large Language Models
  4. 2026Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving
  5. 2025Locality-aware Fair Scheduling in LLM Serving
  6. 2024Ensuring Fair LLM Serving Amid Diverse Applications
  7. 2026Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures
  8. 2026EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference