mlsanityData to 7 Oct 2026

Human-Less LLM Serving: Quantifying the Human Tax on Throughput

arXiv 2606.20577 ↗First posted 3 May 2026cs.NI cs.AI cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Every major LLM serving system is designed to meet TTFT and TPOT SLOs. These metrics capture latency as a human user perceives it, and the mechanisms built to satisfy them are now standard infrastructure. We observe that long-horizon AI tasks call LLMs programmatically in tight loops where no human observes TTFT or TPOT. We ask: how much throughput do serving systems sacrifice to meet TTFT and TPOT SLAs that these workloads never need? We conduct a systematic measurement study across chunk sizes, SLO settings, context lengths, and concurrency levels. We find that the human tax on throughput grows substantially with context length and lands in the 60-93% range. At 64K token contexts, tightening the TTFT SLO to production-typical settings costs a large fraction of throughput versus the human-less baseline. The human tax is larger at higher concurrency and is qualitatively similar across SGLang and Sarathi-Serve. We term the unconstrained optimum human-less serving and provide a prototype demonstrating that it is practical on real workloads. Our findings argue that serving systems should expose workload-class-aware SLA configurations rather than silently applying the human tax uniformly to all traffic.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 17 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

11 of 17 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines8 Aug
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 7

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  7. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  2. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels
  3. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  4. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving
  5. 2026CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving
  6. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production
  7. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
  8. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency