mlsanityData to 14 Oct 2026

LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference

arXiv 2010.13103 ↗First posted 25 Oct 2020cs.DC cs.AR cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

In cloud ML inference systems, batching is an essential technique to increase throughput which helps optimize total-cost-of-ownership. Prior graph batching combines the individual DNN graphs into a single one, allowing multiple inputs to be concurrently executed in parallel. We observe that the coarse-grained graph batching becomes suboptimal in effectively handling the dynamic inference request traffic, leaving significant performance left on the table. This paper proposes LazyBatching, an SLA-aware batching system that considers both scheduling and batching in the granularity of individual graph nodes, rather than the entire graph for flexible batching. We show that LazyBatching can intelligently determine the set of nodes that can be efficiently batched together, achieving an average 15x, 1.5x, and 5.5x improvement than graph batching in terms of average response time, throughput, and SLA satisfaction, respectively.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 7; 2024: 2; 2025: 4; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  2. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 51
  3. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 23
  4. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
  5. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  6. 2022Multi-user Co-inference with Batch Processing Capable Edge Servercited by 11
  7. 2025Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architecturescited by 8
  8. 2025Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilerscited by 7
  9. 2022Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUscited by 6
  10. 2023MOSEL: Inference Serving Using Dynamic Modality Selectioncited by 5
  11. 2021Multi-model Machine Learning Inference Serving with GPU Spatial Partitioningcited by 4
  12. 2023SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platformscited by 4
10 more
  1. 2022KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resourcescited by 3
  2. 2022PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Serverscited by 2
  3. 2021GPU Domain Specialization via Composable On-Package Architecturecited by 1
  4. 2025SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Servicescited by 1
  5. 2023Adaptive Scheduling for Edge-Assisted DNN Servingcited by 0
  6. 2023A Survey of Serverless Machine Learning Model Inferencecited by 0
  7. 2024PREBA: A Hardware/Software Co-Design for Multi-Instance GPU based AI Inference Serverscited by 0
  8. 2025Harmonia: End-to-End RAG Serving Optimizationcited by 0
  9. 2026Mugi: Value Level Parallelism For Efficient LLMscited by 0
  10. 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefillcited by 0