mlsanityData to 7 Oct 2026

Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving

arXiv 2406.13511 ↗First posted 19 Jun 2024cs.DCcited by 13

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) iteratively generate text token by token, with memory usage increasing with the length of generated token sequences. Since the request generation length is generally unpredictable, it is difficult to estimate the time and memory required to process requests, thus posing a challenge for effective request scheduling. Conventional sequence-level scheduling (SLS) serves requests in a first-come first-served (FCFS) manner with static batching where requests with short generation lengths are delayed until those with long ones have finished generation. Besides, to avoid out-of-memory (OOM) errors, SLS batches requests using a small batch size, which limits throughput. Recently proposed iteration-level scheduling (ILS) improves this with continuous batching, timely completing requests and dynamically adding new ones, but often limits the number of parallel-processing requests to OOM errors, thus compromising throughput. Moreover, both SLS and ILS fail to effectively balance workload across multiple LLM instances. To tackle these challenges, we propose slice-level scheduling (SCLS). By splitting the predefined maximal generation length limit into slices and serving batches slice by slice, it provides a precise range of serving time and memory usage for batched requests, laying the foundation for effective scheduling. Experiments confirm that compared with SLS and ILS schedulers, SCLS can improve throughput by up to 315.8% and greatly mitigate load imbalance with proposed batching and offloading algorithms.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 63 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

41 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inference2 Feb
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023Fairness in Serving Large Language Models31 Dec
  5. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  6. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
23 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  3. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  4. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  8. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  9. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  10. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  11. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  12. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  13. 2023Speculative Decoding with Big Little Decodercited by 163
  14. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  15. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  16. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  17. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  18. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  19. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  20. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  21. 2021From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compressioncited by 18
  22. 2023One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Modelscited by 16
  23. 2021Serverless Data Science -- Are We There Yet? A Case Study of Model Servingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 2.

13 papers.

Sort:

By area:

  1. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  2. 2024Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interfacecited by 15
  3. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  4. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  5. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  6. 2025DeServe: Towards Affordable Offline LLM Inference via Decentralizationcited by 4
  7. 2025High-Throughput LLM inference on Heterogeneous Clusterscited by 2
  8. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 1
  9. 2025STAR: Decode-Phase Rescheduling for LLM Inferencecited by 1
  10. 2025Mitigating KV Cache Competition to Enhance User Experience in LLM Inferencecited by 0
  11. 2025WindVE: Collaborative CPU-NPU Vector Embeddingcited by 0
  12. 2026SCALE: Scalable Cross-Attention Learning with Extrapolation for Agentic Workflow Schedulingcited by 0
1 more
  1. 2026End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Servicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
  2. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  3. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  4. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees
  5. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor
  6. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  7. 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference ↗
  8. 2024Efficient LLM Scheduling by Learning to Rank