mlsanityData to 13 Oct 2026

FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism

arXiv 2603.16514 ↗First posted 17 Mar 2026cs.DCcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Modern LLM GPU fleets are provisioned for worst-case context lengths that the vast majority of requests never approach, wasting GPU capacity on idle KV-cache slots. We present FleetOpt, a framework that starts from first principles: given a workload's prompt-length CDF and a P99 TTFT target, derive the minimum-cost fleet analytically, then deploy it in practice. The analytical core models each pool as an M/G/c queue and derives that the minimum-cost fleet is a two-pool architecture -- a short-context pool and a long-context pool -- with an optimal boundary B* satisfying an equal marginal GPU cost condition across both pools. The fundamental barrier to achieving B* is the cost cliff: a hard routing step where requests just above B* consume 8x--42x more GPU capacity than requests just below it (depending on the context window ratio), creating a structural disincentive to lower the boundary. Compress-and-Route (C&R) is the implementation mechanism that resolves this barrier. Gateway-layer extractive compression trims borderline requests below B* before the engine ever sees them, converting the hard hardware boundary into a software parameter read from the workload CDF. The two components are unified in the FleetOpt offline planner: given a CDF and SLO, it returns the optimal (n_s*, n_l*, B*, gamma*) in under 1 ms. On three production traces, the combined framework reduces total GPU cost by 6--82% versus a homogeneous fleet, with C&R contributing 1--44 percentage points beyond plain pool routing depending on workload archetype. The analytical model is validated against a discrete-event simulator (inference-fleet-sim) with <= 3% error on predicted GPU utilization across all pools and workloads.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 22 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

14 of 22 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference13 Mar
  2. 2026LAPS: A Length-Aware-Prefill LLM Serving System4 Jan
  3. 2025CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing22 Dec
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar

Other references · 10

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,274
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 545
  4. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 458
  5. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 186
  6. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  7. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 116
  8. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 65
  9. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 12
  10. 2024PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inferencecited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 5
  2. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  3. 2026Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency
  2. 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
  3. 2024Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
  4. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  5. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  6. 2026Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving
  7. 2025RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference ↗
  8. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices