Predictable LLM Serving on GPU Clusters
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Latency-sensitive inference on shared A100 clusters often suffers noisy-neighbor interference on the PCIe fabric, inflating tail latency and SLO violations. We present a fabric-agnostic, VM-deployable host-level controller that combines dynamic Multi-Instance GPU (MIG) reconfiguration, PCIe-aware placement, and lightweight guardrails (MPS quotas, cgroup I/O). It samples per-tenant tails and system signals, uses topology hints to avoid PCIe hot spots, and gates actions with dwell/cool-down to avoid thrash. On a single host and a 2-node (16-GPU) cluster, SLO miss-rate is reduced by (≈)32% ((≈)1.5) and p99 latency improves (≈)15% with (≤)5% throughput cost versus static MIG and naive placement; ablations show MIG and placement contribute comparably. We also evaluate LLM serving with vLLM on OLMo 2 7B Instruct: TTFT p99 improves (≈)10--15% at (≤)5% cost without changing the controller.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 5 of its 11 references that are in the map. First posted on arXiv 27 Aug 2025.
Builds on
5 of 11 references.
References · 5
- 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
- 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
- 2022MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learningcited by 15
- 2024Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfigurationcited by 3
- 2024Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scalingcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
- 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
- 2026RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving
- 2026The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency
- 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
- 2026Measurement-Driven Diagnosis and Mitigation of Host-CPU Co-location Interference in Single-GPU LLM Serving on a Multi-GPU Server ↗
- 2026Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
- 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing