Data to 7 Oct 2026

Efficient LLM Scheduling by Learning to Rank

arXiv 2408.15792 ↗First posted 28 Aug 2024cs.LGcited by 83

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

In Large Language Model (LLM) inference, the output length of an LLM request is typically regarded as not known a priori. Consequently, most LLM serving systems employ a simple First-come-first-serve (FCFS) scheduling strategy, leading to Head-Of-Line (HOL) blocking and reduced throughput and service quality. In this paper, we reexamine this assumption -- we show that, although predicting the exact generation length of each request is infeasible, it is possible to predict the relative ranks of output lengths in a batch of requests, using learning to rank. The ranking information offers valuable guidance for scheduling requests. Building on this insight, we develop a novel scheduler for LLM inference and serving that can approximate the shortest-job-first (SJF) schedule better than existing approaches. We integrate this scheduler with the state-of-the-art LLM serving system and show significant performance improvement in several important applications: 2.8x lower latency in chatbot serving and 6.5x higher throughput in synthetic data generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 50 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

23 of 50 references.

Closest ancestors · 7

Same direction, newest first

Other references · 16

2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
2018Learning to Optimize Tensor Programscited by 138
2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106

4 more

2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
2021NeuralNDCG: Direct Optimisation of a Ranking Metric via Differentiable Relaxation of Sortingcited by 39
2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 29; 2026: 45.

83 papers.

Sort:

By area:

2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34
2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 17
2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraintscited by 16
2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 15

38 more

2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decodingcited by 14
2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 12
2024Fast Inference for Augmented Large Language Modelscited by 10
2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
2025On Evaluating Performance of LLM Inference Serving Systemscited by 9
2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
2025Adaptively Robust LLM Inference Optimization under Prediction Uncertaintycited by 8
2025LLM Serving Optimization with Variable Prefill and Decode Lengthscited by 7
2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 7
2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inferencecited by 6
2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6
2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 5
2026TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraintscited by 5
2025Queueing, Predictions, and LLMs: Challenges and Open Problemscited by 4
2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
2025Evaluating the Efficacy of LLM-Based Reasoning for Multiobjective HPC Job Schedulingcited by 3
2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Servingcited by 3
2026Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributionscited by 3
2025Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latencycited by 2
2025Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rankcited by 2
2025Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pamperingcited by 2
2026Queue Length Regret Bounds for Contextual Queueing Banditscited by 2
2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
2026Predictive Scheduling for Efficient Inference-Time Reasoning in Large Language Modelscited by 2
2026Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Schedulingcited by 2
2026Agentic AI Systems Should Be Designed as Marginal Token Allocatorscited by 2
2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
2024TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applicationscited by 1
2025Semantic Scheduling for LLM Inferencecited by 1
2025Efficient Serving of LLM Applications with Probabilistic Demand Modelingcited by 1
2025STAR: Decode-Phase Rescheduling for LLM Inferencecited by 1
2025CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancingcited by 1
2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inferencecited by 1
2026Multi-Layer Scheduling for MoE-Based LLM Reasoningcited by 1
2026RLHFless: Serverless Computing for Efficient RLHFcited by 1

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way