SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services
Not on the map. Why?
Abstract (arXiv)
For servers incorporating parallel computing resources, batching is a pivotal technique for providing efficient and economical services at scale. Parallel computing resources exhibit heightened computational and energy efficiency when operating with larger batch sizes. However, in the realm of online services, the adoption of a larger batch size may lead to longer response times. This paper aims to provide a dynamic batching scheme that delicately balances latency and efficiency. The system is modeled as a batch service queue with size-dependent service times. Then, the design of dynamic batching is formulated as a semi-Markov decision process (SMDP) problem, with the objective of minimizing the weighted sum of average response time and average power consumption. A method is proposed to derive an approximate optimal SMDP solution, representing the chosen dynamic batching policy. By introducing an abstract cost to reflect the impact of "tail" states, the space complexity and the time complexity of the procedure can decrease by 63.5% and 98%, respectively. Numerical results showcase the superiority of SMDP-based batching policies across various parameter setups. Additionally, the proposed scheme exhibits noteworthy flexibility in balancing power consumption and latency.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 11 of its 54 references that are in the map. First posted on arXiv 4 Jan 2025.
Builds on
11 of 54 references.
References · 11
- 2014Going Deeper with Convolutionscited by 4,231
- 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
- 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
- 2019Benchmarking TPU, GPU, and CPU Platforms for Deep Learningcited by 51
- 2020LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inferencecited by 21
- 2018Latency and Throughput Characterization of Convolutional Neural Networks for Mobile Computer Visioncited by 13
- 2022Multi-user Co-inference with Batch Processing Capable Edge Servercited by 11
- 2019Queueing Analysis of GPU-Based Inference Servers with Dynamic Batching: A Closed-Form Characterizationcited by 4
- 2023SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platformscited by 4
- 2020DBS: Dynamic Batch Size For Distributed Deep Neural Network Trainingcited by 3
- 2023Packrat: Automatic Reconfiguration for Latency Minimization in CPU-based DNN Servingcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.