mlsanityData to 14 Oct 2026

ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor

arXiv 2505.09142 ↗First posted 14 May 2025cs.DC cs.AI cs.LGcited by 9

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

We propose ELIS, a serving system for Large Language Models (LLMs) featuring an Iterative Shortest Remaining Time First (ISRTF) scheduler designed to efficiently manage inference tasks with the shortest remaining tokens. Current LLM serving systems often employ a first-come-first-served scheduling strategy, which can lead to the "head-of-line blocking" problem. To overcome this limitation, it is necessary to predict LLM inference times and apply a shortest job first scheduling strategy. However, due to the auto-regressive nature of LLMs, predicting the inference latency is challenging. ELIS addresses this challenge by training a response length predictor for LLMs using the BGE model, an encoder-based state-of-the-art model. Additionally, we have devised the ISRTF scheduling strategy, an optimization of shortest remaining time first tailored to existing LLM iteration batching. To evaluate our work in an industrial setting, we simulate streams of requests based on our study of real-world user LLM serving trace records. Furthermore, we implemented ELIS as a cloud-native scheduler system on Kubernetes to evaluate its performance in production environments. Our experimental results demonstrate that ISRTF reduces the average job completion time by up to 19.6%.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 38 references that are in the map. First posted on arXiv 14 May 2025.

Builds on

24 of 38 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  2. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  7. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
  8. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  6. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  8. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  9. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  10. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  11. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  12. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
4 more
  1. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  2. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  3. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  4. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Modelscited by 10
  2. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 8
  3. 2026Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributionscited by 3
  4. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2
  5. 2025BeLLMan: Controlling LLM Congestioncited by 0
  6. 2026Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUscited by 0
  7. 2026NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placementcited by 0
  8. 2026LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithmcited by 0
  9. 2026Jumping the Line: Exploiting Length Predictions in LLM Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient LLM Scheduling by Learning to Rank
  2. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  3. 2025Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
  4. 2024Multi-Bin Batching for Increasing LLM Inference Throughput
  5. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  6. 2026EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference
  7. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inference
  8. 2025Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism