mlsanityData to 14 Oct 2026

BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving

arXiv 2507.17120 ↗First posted 23 Jul 2025cs.DC cs.AIcited by 5

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) have become increasingly popular in various areas, traditional business gradually shifting from rule-based systems to LLM-based solutions. However, the inference of LLMs is resource-intensive or latency-sensitive, posing significant challenges for serving systems. Existing LLM serving systems often use static or continuous batching strategies, which can lead to inefficient GPU memory utilization and increased latency, especially under heterogeneous workloads. These methods may also struggle to adapt to dynamic workload fluctuations, resulting in suboptimal throughput and potential service level objective (SLO) violations. In this paper, we introduce BucketServe, a bucket-based dynamic batching framework designed to optimize LLM inference performance. By grouping requests into size-homogeneous buckets based on sequence length, BucketServe minimizes padding overhead and optimizes GPU memory usage through real-time batch size adjustments preventing out-of-memory (OOM) errors. It introduces adaptive bucket splitting/merging and priority-aware scheduling to mitigate resource fragmentation and ensure SLO compliance. Experiment shows that BucketServe significantly outperforms UELLM in throughput, achieving up to 3.58x improvement. It can also handle 1.93x more request load under the SLO attainment of 80% compared with DistServe and demonstrates 1.975x higher system load capacity compared to the UELLM.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 36 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

29 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  3. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
11 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  3. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  4. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  5. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  6. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  7. 2023From Sparse to Dense: GPT-4 Summarization with Chain of Density Promptingcited by 52
  8. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  9. 2024DRPC: Distributed Reinforcement Learning Approach for Scalable Resource Provisioning in Container-based Clusterscited by 4
  10. 2024StatuScale: Status-aware and Elastic Scaling Strategy for Microservice Applicationscited by 2
  11. 2023Resource Management for GPT-based Model Deployed on Clouds: Challenges, Solutions, and Future Directionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026LAPS: A Length-Aware-Prefill LLM Serving Systemcited by 4
  2. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  3. 2026Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inferencecited by 0
  4. 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefillcited by 0
  5. 2026Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Bin Batching for Increasing LLM Inference Throughput
  2. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  3. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  4. 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications
  5. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  6. 2025CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing
  7. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction
  8. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving