SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Serving large language models under latency service-level objectives (SLOs) is a configuration-heavy systems problem with an unusually failure-prone search space: many plausible configurations crash outright or miss user-visible latency targets, and standard black-box optimizers treat these failures as wasted trials. We present SLO-Guard, a crash-aware autotuner for vLLM serving that treats crashes as first-class observations. SLO-Guard combines a feasible-first Thermal Budget Annealing (TBA) exploration phase with a warm-started Tree-structured Parzen Estimator (TPE) exploitation phase; the handoff replays all exploration history, including crashes encoded as extreme constraint violations. We additionally contribute a configuration-repair pass, a GPU-aware KV-cache memory guard, and a four-category crash taxonomy. We evaluate SLO-Guard on Qwen2-1.5B served with vLLM 0.19 on an NVIDIA A100 40GB. Across a pre-specified five-seed study, both SLO-Guard and uniform random search attain 75/75 feasibility with zero crashes under the corrected concurrent harness, and are statistically tied on best-achieved latency (Mann-Whitney two-sided p=0.84). SLO-Guard's advantage is in budget consistency: more trials in the fast-serving regime (10.20 vs. 7.40 out of 15; one-sided p=0.014) and higher post-handoff consistency (0.876 vs. 0.539; p=0.010). Under concurrent load, SLO-Guard's cross-seed standard deviation on best latency is 4.4x tighter than random search's (2.26 ms vs. 10.00 ms). A harness-replication analysis shows that the consistency findings survive an independent sequential-dispatch measurement condition. The central claim is not that SLO-Guard finds a better final configuration, but that it spends a fixed tuning budget more predictably once the fast regime has been found.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 10 of its 25 references that are in the map. First posted on arXiv 19 Apr 2026.
Builds on
10 of 25 references.
Closest ancestors · 2
Same direction, newest first
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
- 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
Other references · 8
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
- 2019MLPerf Inference Benchmarkcited by 199
- 2014Bayesian Optimization with Unknown Constraintscited by 109
- 2017Constrained Bayesian Optimization with Noisy Experimentscited by 104
- 2022Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningcited by 20
- 2024Local Bayesian Optimization for Controller Tuning with Crash Constraintscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration
- 2026LUMEN: Coordinated Failure Recovery for Distributed LLM Serving
- 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
- 2025Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms
- 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving
- 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing
- 2024EconoServe: Maximizing Multi-Resource Utilization with SLO Guarantees in LLM Serving
- 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications