# Predictable LLM Serving on GPU Clusters

arXiv: https://arxiv.org/abs/2508.20274 · HTML: https://mlsanity.com/paper/2508.20274/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 5 of 5)

- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Themis: Fair and Efficient GPU Cluster Scheduling](https://mlsanity.com/paper/1907.01484/) (2019)
- [MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learning](https://mlsanity.com/paper/2207.11428/) (2022)
- [Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration](https://mlsanity.com/paper/2407.13126/) (2024)
- [Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling](https://mlsanity.com/paper/2404.00704/) (2024)

## Abstract (arXiv)

Latency-sensitive inference on shared A100 clusters often suffers noisy-neighbor interference on the PCIe fabric, inflating tail latency and SLO violations. We present a fabric-agnostic, VM-deployable host-level controller that combines dynamic Multi-Instance GPU (MIG) reconfiguration, PCIe-aware placement, and lightweight guardrails (MPS quotas, cgroup I/O). It samples per-tenant tails and system signals, uses topology hints to avoid PCIe hot spots, and gates actions with dwell/cool-down to avoid thrash. On a single host and a 2-node (16-GPU) cluster, SLO miss-rate is reduced by (≈)32% ((≈)1.5) and p99 latency improves (≈)15% with (≤)5% throughput cost versus static MIG and naive placement; ablations show MIG and placement contribute comparably. We also evaluate LLM serving with vLLM on OLMo 2 7B Instruct: TTFT p99 improves (≈)10--15% at (≤)5% cost without changing the controller.