# SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services

arXiv: https://arxiv.org/abs/2501.02181 · HTML: https://mlsanity.com/paper/2501.02181/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 54 references that are in the map.
- First posted on arXiv 4 Jan 2025.

## Builds on (most cited first, 11 of 11)

- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training](https://mlsanity.com/paper/2208.06102/) (2022)
- [Benchmarking TPU, GPU, and CPU Platforms for Deep Learning](https://mlsanity.com/paper/1907.10701/) (2019)
- [LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference](https://mlsanity.com/paper/2010.13103/) (2020)
- [Latency and Throughput Characterization of Convolutional Neural Networks for Mobile Computer Vision](https://mlsanity.com/paper/1803.09492/) (2018)
- [Multi-user Co-inference with Batch Processing Capable Edge Server](https://mlsanity.com/paper/2206.06304/) (2022)
- [Queueing Analysis of GPU-Based Inference Servers with Dynamic Batching: A Closed-Form Characterization](https://mlsanity.com/paper/1912.06322/) (2019)
- [SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms](https://mlsanity.com/paper/2301.12865/) (2023)
- [DBS: Dynamic Batch Size For Distributed Deep Neural Network Training](https://mlsanity.com/paper/2007.11831/) (2020)
- [Packrat: Automatic Reconfiguration for Latency Minimization in CPU-based DNN Serving](https://mlsanity.com/paper/2311.18174/) (2023)

## Built on by (most cited first, 1 of 1)

- [Optimal Expert Selection for Distributed Mixture-of-Experts at the Wireless Edge](https://mlsanity.com/paper/2503.13421/) (2025)

## Abstract (arXiv)

For servers incorporating parallel computing resources, batching is a pivotal technique for providing efficient and economical services at scale. Parallel computing resources exhibit heightened computational and energy efficiency when operating with larger batch sizes. However, in the realm of online services, the adoption of a larger batch size may lead to longer response times. This paper aims to provide a dynamic batching scheme that delicately balances latency and efficiency. The system is modeled as a batch service queue with size-dependent service times. Then, the design of dynamic batching is formulated as a semi-Markov decision process (SMDP) problem, with the objective of minimizing the weighted sum of average response time and average power consumption. A method is proposed to derive an approximate optimal SMDP solution, representing the chosen dynamic batching policy. By introducing an abstract cost to reflect the impact of "tail" states, the space complexity and the time complexity of the procedure can decrease by 63.5% and 98%, respectively. Numerical results showcase the superiority of SMDP-based batching policies across various parameter setups. Additionally, the proposed scheme exhibits noteworthy flexibility in balancing power consumption and latency.