mlsanityData to 14 Oct 2026

Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction

arXiv 2404.08509 ↗First posted 12 Apr 2024cs.DC cs.CL cs.LGcited by 66

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 30 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

11 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
  2. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 9

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  4. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  5. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  6. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  7. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  8. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  9. 2021The Case for Task Sampling based Learning for Cluster Job Schedulingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 22; 2026: 31.

66 papers.

Sort:

By area:

  1. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  2. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 109
  3. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  4. 2024Queue management for slo-oriented large language model servingcited by 36
  5. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 35
  6. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  7. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 29
  8. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  9. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 20
  10. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 19
  11. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 19
  12. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
38 more
  1. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Managementcited by 16
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  3. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
  4. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11
  5. 2024Fast Inference for Augmented Large Language Modelscited by 10
  6. 2025Adaptively Robust LLM Inference Optimization under Prediction Uncertaintycited by 10
  7. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  8. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 9
  9. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  10. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
  11. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 8
  12. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 8
  13. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  14. 2025Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Expertscited by 7
  15. 2025LLM Serving Optimization with Variable Prefill and Decode Lengthscited by 7
  16. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Servingcited by 6
  17. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inferencecited by 6
  18. 2025Queueing, Predictions, and LLMs: Challenges and Open Problemscited by 5
  19. 2025Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latencycited by 4
  20. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
  21. 2026Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributionscited by 3
  22. 2025Equinox: Holistic Fair Scheduling in Serving Large Language Modelscited by 2
  23. 2025Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rankcited by 2
  24. 2025Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pamperingcited by 2
  25. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
  26. 2026Coordinated Scheduling for MoE LLM Servingcited by 2
  27. 2026EnerInfer: Energy-Aware On-Device LLM Inferencecited by 2
  28. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  29. 2024TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applicationscited by 1
  30. 2025Efficient Serving of LLM Applications with Probabilistic Demand Modelingcited by 1
  31. 2025STAR: Decode-Phase Rescheduling for LLM Inferencecited by 1
  32. 2025CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancingcited by 1
  33. 2026Multi-Layer Scheduling for MoE-Based LLM Reasoningcited by 1
  34. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
  35. 2026Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM poolscited by 1
  36. 2026OUTLETS: Output-Length Prediction from Speculative Decoding Backbonescited by 1
  37. 2026SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybriditycited by 0
  38. 2026Duration Aware Scheduling for ASR Serving Under Workload Driftcited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  2. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
  3. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
  4. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  5. 2026EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference
  6. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  8. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving