mlsanityData to 14 Oct 2026

FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration

arXiv 2608.19659 ↗First posted 20 Aug 2026cs.LG cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Choosing tensor-parallel (TP) degrees and replica counts for an LLM serving fleet is difficult because performance is not monotonic in TP and the feasible choice can change with load. Exhaustive profiling resolves this uncertainty, but measures many configurations that do not affect the final resource allocation. We present FleetSieve, which selects measurements according to their expected effect on a resource-coupled, SLO-aware fleet decision. FleetSieve models capacity and tail latency jointly, compares conservative and optimistic allocations, and stops when their remaining decision gap is below a specified tolerance. On a fixed H100 measurement grid for a 31B-parameter open-weight model, FleetSieve reaches the oracle aggregate decision using 22,200 GPU-seconds, 6.9% less than uniform random profiling in the fixed comparison. Across 200 random reveal orders, its mean saving over random profiling is 5.4% (95% bootstrap CI: 3.5-7.2%). The fixed-comparison saving is 21.5% for Chat, while FleetSieve does not use the fewest GPU-seconds for Code. Joint capacity and tail modeling also avoids selecting a configuration whose 46.4-second completion p99 violates a 30-second SLO. In a 16-GPU allocation, an incorrect sparse-profile decision loses up to 1.93 requests/s and 12.4 percentage points of max-min fulfillment. Boundary repeats and BurstGPT measurements support the observed load-dependent tail-latency mechanism.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 15 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

11 of 15 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism6 May
  2. 2024SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines8 Aug
  3. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 7

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  3. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  4. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  5. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  6. 2021Targeted Active Learning for Bayesian Decision-Makingcited by 12
  7. 2023A Fast Algorithm for the Real-Valued Combinatorial Pure Exploration of Multi-Armed Banditcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
  2. 2026Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels
  3. 2026Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking ↗
  4. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
  5. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  6. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  7. 2025Predictable LLM Serving on GPU Clusters
  8. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration