mlsanityData to 7 Oct 2026

LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

arXiv 2608.08382 ↗First posted 9 Aug 2026cs.AI cs.DCcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control. Enabling fractional sharing of the inference engine requires a real-time, per-request attribution primitive that is accurate and light enough to run inside the scheduling loop. We present LLMVisor, a roofline-guided latency attribution model that captures the memory-bound and compute-bound phases via a concise piecewise-linear form over features proportional to FLOPs and memory I/O traffic. LLMVisor decomposes batch latency into additive, per-request shares and runs efficiently at microsecond scale. We evaluate LLMVisor across Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 GPUs under varying tensor parallelism and workload mixes. Compared to a token-count baseline, LLMVisor attains near-perfect R-squared and reduces relative error by up to 2.5x and 3.3x at p90 and p99, respectively, for prefill, and by up to 3.5x and 4.4x for decode, despite batching variability and sequence divergence.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 22 references that are in the map. First posted on arXiv 9 Aug 2026.

Builds on

12 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  2. 2023Fairness in Serving Large Language Models31 Dec

Other references · 10

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  5. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  6. 2024Eagle: Efficient Training-Free Router for Multi-LLM Inferencecited by 34
  7. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  8. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 12
  9. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 10
  10. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Billscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  2. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning
  3. 2025Cronus: Efficient LLM inference on Heterogeneous GPU Clusters via Partially Disaggregated Prefill
  4. 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
  5. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  6. 2026Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices
  7. 2026Lodestar: An Online-Learning LLM Inference Router
  8. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains