# Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching

arXiv: https://arxiv.org/abs/2503.05248 · HTML: https://mlsanity.com/paper/2503.05248/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2025.

## Built on by (most cited first, 8 of 8)

- [Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity](https://mlsanity.com/paper/2505.14884/) (2025)
- [RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU](https://mlsanity.com/paper/2504.15302/) (2025)
- [AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimization](https://mlsanity.com/paper/2508.01744/) (2025)
- [Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling](https://mlsanity.com/paper/2507.18006/) (2025)
- [Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks](https://mlsanity.com/paper/2603.05692/) (2026)
- [Streaming Tensor Programs: A Streaming Abstraction for Dynamic Parallelism](https://mlsanity.com/paper/2511.07776/) (2025)
- [Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference](https://mlsanity.com/paper/2605.06046/) (2026)
- [Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice](https://mlsanity.com/paper/2606.22327/) (2026)

## Abstract (arXiv)

The increasing adoption of large language models (LLMs) necessitates inference serving systems that can deliver both high throughput and low latency. Deploying LLMs with hundreds of billions of parameters on memory-constrained GPUs exposes significant limitations in static batching methods. Current inference serving systems often treat batch sizes as fixed hyper-parameters, hindering real-time adaptation to varying system conditions. In this paper, we propose a dynamic batching method that continuously monitors memory utilization and adheres to service-level agreements (SLAs) to enable real-time batch size configuration adjustment. The method comprises two core components: a memory-aware batch scheduler that dynamically allocates GPU resources and a latency feedback mechanism that optimizes decoding processes under SLA constraints. The numerical experiments demonstrate throughput gains of 8% to 28% and capacity improvements of 22% compared to traditional static batching methods, while maintaining full compatibility with existing inference infrastructure. These results highlight the effectiveness of dynamic batching in balancing computational efficiency and quality-of-service requirements for contemporary LLM deployment scenarios. The source code of this work is publicly available at https://github.com/KevinLee1110/dynamic-batching.