# LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference

arXiv: https://arxiv.org/abs/2010.13103 · HTML: https://mlsanity.com/paper/2010.13103/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2020.

## Built on by (most cited first, 22 of 22)

- [OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization](https://mlsanity.com/paper/2304.07493/) (2023)
- [ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization](https://mlsanity.com/paper/2208.14286/) (2022)
- [vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training](https://mlsanity.com/paper/2312.12391/) (2023)
- [Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving](https://mlsanity.com/paper/2312.05385/) (2023)
- [Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation](https://mlsanity.com/paper/2405.01814/) (2024)
- [Multi-user Co-inference with Batch Processing Capable Edge Server](https://mlsanity.com/paper/2206.06304/) (2022)
- [Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures](https://mlsanity.com/paper/2504.11750/) (2025)
- [Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers](https://mlsanity.com/paper/2509.11173/) (2025)
- [Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs](https://mlsanity.com/paper/2209.13443/) (2022)
- [MOSEL: Inference Serving Using Dynamic Modality Selection](https://mlsanity.com/paper/2310.18481/) (2023)
- [Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning](https://mlsanity.com/paper/2109.01611/) (2021)
- [SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms](https://mlsanity.com/paper/2301.12865/) (2023)
- [KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources](https://mlsanity.com/paper/2210.05889/) (2022)
- [PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Servers](https://mlsanity.com/paper/2202.13481/) (2022)
- [GPU Domain Specialization via Composable On-Package Architecture](https://mlsanity.com/paper/2104.02188/) (2021)
- [SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services](https://mlsanity.com/paper/2501.02181/) (2025)
- [Adaptive Scheduling for Edge-Assisted DNN Serving](https://mlsanity.com/paper/2304.09961/) (2023)
- [A Survey of Serverless Machine Learning Model Inference](https://mlsanity.com/paper/2311.13587/) (2023)
- [PREBA: A Hardware/Software Co-Design for Multi-Instance GPU based AI Inference Servers](https://mlsanity.com/paper/2411.19114/) (2024)
- [Harmonia: End-to-End RAG Serving Optimization](https://mlsanity.com/paper/2505.07833/) (2025)
- [Mugi: Value Level Parallelism For Efficient LLMs](https://mlsanity.com/paper/2601.10823/) (2026)
- [ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill](https://mlsanity.com/paper/2606.22541/) (2026)

## Abstract (arXiv)

In cloud ML inference systems, batching is an essential technique to increase throughput which helps optimize total-cost-of-ownership. Prior graph batching combines the individual DNN graphs into a single one, allowing multiple inputs to be concurrently executed in parallel. We observe that the coarse-grained graph batching becomes suboptimal in effectively handling the dynamic inference request traffic, leaving significant performance left on the table. This paper proposes LazyBatching, an SLA-aware batching system that considers both scheduling and batching in the granularity of individual graph nodes, rather than the entire graph for flexible batching. We show that LazyBatching can intelligently determine the set of nodes that can be efficiently batched together, achieving an average 15x, 1.5x, and 5.5x improvement than graph batching in terms of average response time, throughput, and SLA satisfaction, respectively.