mlsanityData to 7 Oct 2026

A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality

arXiv 2609.18005 ↗First posted 16 Sep 2026cs.CL cs.LGcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model optimization is an active research area, spanning quantization of model weights, early-exit methods for skipping layers, and speculative decoding. Each track uses its own quality measures, typically an idiosyncratic benchmark score. Few approach the measurement precision required by other scientific disciplines. We propose a rigorous methodology for measuring output quality, suitable for cross-system and cross-technique comparison. We score outputs with an LLM as a judge, but calibrate the judge formally: we compare its scores on two ordinary runs of a model given the same prompts, verifying that it shows no systematic preference between statistically equivalent outputs and measuring its per-sample noise. Each design also includes a 'null' condition, provably identical in distribution to the unmodified model, whose measured difference must be zero. With this one instrument we measure several acceleration techniques on the same prompts, so their quality costs can be compared. Perceived quality proves highly dependent on the domain of discourse. A 4-bit model was indistinguishable from its 16-bit original down to our design's +/-0.3-point resolution, in English prose and Chinese alike. At 3-bit precision the same prompts lost 0.5 points in English prose, 0.9 in Chinese, and 1.1 on multi-step math; early exit that cost 0.7 points on prose cost 2.5 on math, cutting correctly solved problems from 19 of 27 to 6. The pattern held for models from Alibaba and from Meta, but not its magnitude: the same quantizer cost Meta's model 1.8 points where it cost Alibaba's 0.7. A model's certainty about a token predicts how likely it is to differ from the full model's choice, but not how much that difference affects judged quality, so acceptance rules relying on certainty cannot distinguish errors that matter from errors that don't.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 33 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

25 of 33 references.

References · 25

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  4. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  5. 2022Confident Adaptive Language Modelingcited by 297
  6. 2024Massive Activations in Large Language Modelscited by 277
  7. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 130
  8. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  9. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  10. 2024Model Equality Testing: Which Model Is This API Serving?cited by 44
  11. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  12. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
13 more
  1. 2024Accuracy is Not All You Needcited by 27
  2. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  3. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  4. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  5. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  6. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluationcited by 7
  7. 2026LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluationcited by 6
  8. 2026When LLMs get significantly worse: A statistical approach to detect model degradationscited by 5
  9. 2025Reject Only Critical Tokens: Pivot-Aware Speculative Decodingcited by 4
  10. 2026Statistically-Lossless Quantization of Large Language Modelscited by 4
  11. 2026Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLMcited by 2
  12. 2026A Practical Investigation of Training-free Relaxed Speculative Decodingcited by 2
  13. 2025English K_Quantization of LLMs Does Not Disproportionately Diminish Multilingual Performancecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
  2. 2026CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
  3. 2026Probabilistic Calibration Is a Trainable Capability in Language Models
  4. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  5. 2025On Evaluating Performance of LLM Inference Serving Systems
  6. 2026Understanding Tone-Dependent Inference Cost in Large Language Models
  7. 2024The Impact of Inference Acceleration on Bias of LLMs
  8. 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression