mlsanityData to 14 Oct 2026

Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization

arXiv 2502.15763 ↗First posted 14 Feb 2025cs.DC cs.AI cs.ARcited by 2

Not on the map. Why?

Abstract (arXiv)

With the development of large language models (LLMs), it has become increasingly important to optimize hardware usage and improve throughput. In this paper, we study the inference optimization of the serving system that deploys LLMs. To optimize system throughput and maximize hardware utilization, we formulate the inference optimization problem as a mixed-integer programming (MIP) model and propose a hybrid offline-online method as solution. The offline method improves large-scale inference systems by introducing a Minimizing Makespan Bin Packing Problem. We further provide a theoretical lower bound computation method. Then, we propose an online sorting and preemptive scheduling method to better utilize hardware. In the online iteration scheduling process, a Lagrangian method is applied to evaluate the cost efficiency of inserting prefill stages versus decode stages at each iteration and dynamically determine when to preempt decoding tasks and insert prefill tasks. Experiments using real-world data from the LLaMA-65B model and the GSM8K dataset demonstrate that system utilization improves from 80.2% to 89.1%, and the total inference time decreases from 201.00 to 190.58 seconds. A 100-cases study shows that our method consistently outperforms the baseline method and improves the utilization rate by 8.0% on average. Finally, we discuss potential future extensions, including stochastic modeling, reinforcement learning-based schedulers, and dynamic decision-making strategies for system throughput and hardware utilization.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

25 of 38 references.

References · 25

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
13 more
  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  4. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  5. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  8. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  9. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  10. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  11. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  12. 2023Fairness in Serving Large Language Modelscited by 78
  13. 2024SparseLLM: Towards Global Pruning for Pre-trained Language Modelscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approachcited by 0
  2. 2026Threshold-Based Exclusive Batching for LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  2. 2025Online Scheduling for LLM Inference with KV Cache Constraints
  3. 2024Inference Performance Optimization for Large Language Models on CPUs
  4. 2024Multi-Bin Batching for Increasing LLM Inference Throughput
  5. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  6. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
  7. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  8. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems