# GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems

arXiv: https://arxiv.org/abs/2609.14864 · HTML: https://mlsanity.com/paper/2609.14864/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 12 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 4 of 4)

- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems](https://mlsanity.com/paper/2505.11415/) (2025)
- [LIMINAL: Exploring The Frontiers of LLM Decode Performance](https://mlsanity.com/paper/2507.14397/) (2025)
- [RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysis](https://mlsanity.com/paper/2602.11506/) (2026)

## Abstract (arXiv)

We predict single-sequence model throughput from GGUF metadata using roofline-shaped predictors with quantization-specific scale factors fitted on reference models. The scored cohort comprises 318 phase-depth measurements from 53 host-file configurations on two Apple M4 Max systems and an NVIDIA RTX 5080. On host-specific held-out sets of four, five, and two configurations, an active-parameter decode model obtains 13.1%, 14.4%, and 36.1% mean absolute percentage error (MAPE), versus 49.4%, 55.3%, and 51.9% when charging total parameters. Leave-one-host-out coefficients fitted on the other two systems yield 11.6%, 16.8%, and 36.0% test MAPE. A low-bit model ladder changes ordering across runtime stacks. The P2 prefill baseline gives 18.7%, 22.2%, and 108.2% test MAPE. GGUF structure helps on all three systems, but fitted efficiencies are not universal.