mlsanityData to 14 Oct 2026

Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Traces

arXiv 2608.00423 ↗First posted 1 Aug 2026cs.AIcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Production LLM serving generates millions of diverse requests, making full-trace replay across serving configurations increasingly expensive. Existing trace reduction methods mainly preserve workload distributions or representative requests, but bottleneck-revealing workloads may be rare and non-representative. Moreover, evidence for one component cannot compensate for missing evidence in another, while using predicted bottlenecks as target truth creates circular evaluation. These limitations make it necessary to preserve evidence for every bottleneck component rather than rely on workload representativeness alone. We propose Bottleneck-Preserving Witnessing (BPW), a quality-constrained framework for compact and diagnostically reliable LLM serving replay suites. BPW first performs Workload Candidate Nomination using response-blind workload features and closed source-side measurements. This stage identifies workloads that may expose scheduler, prefill, decode, or KV-cache bottlenecks. Coverage-Priority Sequence Construction then organizes multi-component proposals as reusable hyperedges and prioritizes weak and uncovered dimensions. Finally, Bottleneck Truth Verification derives prediction-independent labels solely from direct target-system measurements. The verified results determine the earliest prefix satisfying the direct two-witness requirement for every component. Experiments on BurstGPT, ServeGen, and Mooncake show that BPW reaches the verified gate with a compact workload set and outperforms 16 policies, achieving relative improvements of 2.3% and 16.3% in Mean prefix Macro-F1 and WBRC-AUC, respectively. Stage-resolved and sensitivity analyses confirm the distinct contributions and local stability of its three stages. Our code is publicly available at https://github.com/llmllmllm/BPW

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 31 references that are in the map. First posted on arXiv 1 Aug 2026.

Builds on

20 of 31 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 16

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  3. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  4. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 86
  5. 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
  6. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  7. 2017Why Adaptively Collected Data Have Negative Bias and How to Correct for Itcited by 36
  8. 2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 35
  9. 2025Reliable and Efficient Amortized Model-based Evaluationcited by 31
  10. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 26
  11. 2018Mitigating Bias in Adaptive Data Gathering via Differential Privacycited by 21
  12. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 21
4 more
  1. 2023SuperServe: Fine-Grained Inference Serving for Unpredictable Workloadscited by 12
  2. 2025Faster Configuration Performance Bug Testing with Neural Dual-level Prioritizationcited by 9
  3. 2023Bayesian Active Learning in the Presence of Nuisance Parameterscited by 6
  4. 2024Submodular Information Selection for Hypothesis Testing with Misclassification Penaltiescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Remember Before You're Asked: MemDream for Self-Probing Memory Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
  2. 2026Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing
  3. 2026Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels
  4. 2026Fail-Closed Lowering of Resident KV Claims onto LLM Serving Runtimes
  5. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  6. 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
  7. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  8. 2026TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Serving