mlsanityData to 14 Oct 2026

Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills

arXiv 2609.24991 ↗First posted 21 Sep 2026cs.DC cs.PFcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Organizations pay for AI through disconnected ledgers: Kubernetes allocations for self-hosted inference, gateway logs, and per-token bills from API providers. We present unalloc, an open-source tool that joins OpenCost, LiteLLM, OpenAI and Anthropic cost data into one exact ledger and reports the share of spend with no owner, and use it to study where attribution breaks at the seams between these systems. Five case studies run inference for real or simulate it: a vLLM-style serving simulator with paged KV memory and prefix caching; a PyTorch transformer serving a multi-tenant trace with a real KV cache; tensor- and pipeline-parallel inference on torch.distributed; the unmodified CLI against mock provider APIs; and four downstream use cases. At the seams, in a constructed multi-pod deployment scenario -- one month of synthetic OpenCost allocations, not observed billing data -- owner labels set only on LeaderWorkerSet leader pods leave 66% of that deployment's GPU bill unowned, and the natural fallback key assigns 61% of it to a Helm chart name while the headline unallocated share falls to 4%; enabling every source double counts all gateway spend; and reading one page of a billing API reports a quarter of spend. Inside a shared inference server the metering rule decides who pays: on an NVIDIA H100 running vLLM, a token meter assigns a retrieval-heavy tenant 12-14 percentage points more of the bill than an equal time-share meter at every load tested, while GPU utilization reads 97-99% across configured loads of 2 to 16 requests per second (3.7 to 26.9 completed requests per second; the configured rate counts session-initial arrivals only) and power draw tracks load. Neither meter is a ground truth; we position these results against recent Shapley-based energy attribution. Code, raw data, captured evidence, figures and the paper regenerate from the repository.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 24 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

14 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving9 Aug
  2. 2026Request-Level Energy Attribution for Batched LLM Serving11 Jul
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 10

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2022Efficiently Scaling Transformer Inferencecited by 539
  6. 2024Carbon accounting in the Cloud: a methodology for allocating emissions across data center userscited by 4
  7. 2024ABACUS: A FinOps Service for Cloud Cost Optimizationcited by 4
  8. 2026Preserving Admission Responsibility in Multi-Tenant Large Language Model Prefix Cachescited by 2
  9. 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platformscited by 2
  10. 2026Cost-Governed RAG: Unified Per-Tenant Cost Attribution Across Retrieval and Generation in Multi-Tenant LLM Systemscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Token Management in Multi-Tenant AI Inference Platforms
  2. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management
  3. 2026Position: LLM Inference Should Be Evaluated as Energy-to-Token Production
  4. 2026Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs
  5. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
  6. 2026Total Cost of Agency: Exact Attribution of Memory Injection Cost in Multi-Agent LLM Workflows
  7. 2026Memory Scarcity, Open Models, and the Restructuring of the AI Industry, 2026-2030 -- A quantitative scenario analysis of inference economics, training-cost divergence, and infrastructure solvency
  8. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands