mlsanityData to 14 Oct 2026

GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems

arXiv 2609.14864 ↗First posted 14 Sep 2026cs.AI cs.LG cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

We predict single-sequence model throughput from GGUF metadata using roofline-shaped predictors with quantization-specific scale factors fitted on reference models. The scored cohort comprises 318 phase-depth measurements from 53 host-file configurations on two Apple M4 Max systems and an NVIDIA RTX 5080. On host-specific held-out sets of four, five, and two configurations, an active-parameter decode model obtains 13.1%, 14.4%, and 36.1% mean absolute percentage error (MAPE), versus 49.4%, 55.3%, and 51.9% when charging total parameters. Leave-one-host-out coefficients fitted on the other two systems yield 11.6%, 16.8%, and 36.0% test MAPE. A low-bit model ladder changes ordering across runtime stacks. The P2 prefill baseline gives 18.7%, 22.2%, and 108.2% test MAPE. GGUF structure helps on all three systems, but fitted efficiencies are not universal.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 4 of its 12 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

4 of 12 references.

References · 4

  1. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  2. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  3. 2025LIMINAL: Exploring The Frontiers of LLM Decode Performancecited by 11
  4. 2026RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysiscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  2. 2025Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
  3. 2026Dissecting GPU Utilization for LLM Inference on Nvidia Hopper
  4. 2026Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
  5. 2026Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference
  6. 2026Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
  7. 2026Sustained 70B-Class AWQ Inference on a Single NVIDIA L20: Throughput, Stability, Energy, and Quality Characterization ↗
  8. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures