mlsanityData to 14 Oct 2026

Multi-Bin Batching for Increasing LLM Inference Throughput

arXiv 2412.04504 ↗First posted 3 Dec 2024cs.CL cs.DC cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the inference jobs on servers (e.g. GPUs), enabling the system to maximize throughput by allowing multiple requests to be processed in parallel. However, requests often have varying generation lengths, causing resource underutilization, as hardware must wait for the longest-running request in the batch to complete before moving to the next batch. We formalize this problem from a queueing-theoretic perspective, and aim to design a control policy which is throughput-optimal under a static-batching framework. We propose Multi-Bin Batching, a simple yet effective method that can provably improve LLM inference throughput under this framework by grouping requests with similar (predicted) execution times into predetermined bins. Through a combination of theoretical analysis and experiments, including real-world LLM inference scenarios with static and continuous-batching baselines, we demonstrate that multi-bin batching substantially improves throughput over static batching and quantify the remaining gap to native continuous batching under both oracle and estimated length information.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

29 of 39 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
17 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  3. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  4. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  6. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  7. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  8. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  9. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  10. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  11. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  12. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  13. 2024Queue management for slo-oriented large language model servingcited by 36
  14. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20
  15. 2024A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Lengthcited by 16
  16. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  17. 2019Queueing Analysis of GPU-Based Inference Servers with Dynamic Batching: A Closed-Form Characterizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 4.

9 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 61
  3. 2025Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Modelscited by 10
  4. 2026LAPS: A Length-Aware-Prefill LLM Serving Systemcited by 4
  5. 2025Equinox: Holistic Fair Scheduling in Serving Large Language Modelscited by 2
  6. 2026A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraintscited by 2
  7. 2026DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Trainingcited by 2
  8. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottleneckscited by 1
  9. 2025Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Executioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
  2. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  3. 2025Seesaw: High-throughput LLM Inference via Model Re-sharding
  4. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  5. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  6. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  8. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents