# Certifying Compressed Language Models: An Audit and a Statistical Toolkit

arXiv: https://arxiv.org/abs/2608.15046 · HTML: https://mlsanity.com/paper/2608.15046/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 23 references that are in the map.
- First posted on arXiv 15 Aug 2026.

## Builds on (most cited first, 23 of 23)

- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [On the Impact of Calibration Data in Post-training Quantization and Pruning](https://mlsanity.com/paper/2311.09755/) (2023)
- [Accuracy is Not All You Need](https://mlsanity.com/paper/2407.09141/) (2024)
- [Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs](https://mlsanity.com/paper/2405.20835/) (2024)
- [When LLMs get significantly worse: A statistical approach to detect model degradations](https://mlsanity.com/paper/2602.10144/) (2026)
- [Statistically-Lossless Quantization of Large Language Models](https://mlsanity.com/paper/2605.02404/) (2026)
- [Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation](https://mlsanity.com/paper/2604.27405/) (2026)
- [The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs](https://mlsanity.com/paper/2607.08734/) (2026)
- [Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation](https://mlsanity.com/paper/2605.04135/) (2026)
- [Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment](https://mlsanity.com/paper/2606.19558/) (2026)
- [Mitigating LLM-based p-Hacking by Preregistering for the Next LLM](https://mlsanity.com/paper/2606.27687/) (2026)
- [Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstration](https://mlsanity.com/paper/2607.14399/) (2026)

## Built on by (most cited first, 2 of 2)

- [What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects](https://mlsanity.com/paper/2610.05533/) (2026)
- [What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains](https://mlsanity.com/paper/2610.19498/) (2026)

## Abstract (arXiv)

A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original. That quantity is least informative when two models are most alike: a net delta is what survives cancellation between opposing per-item changes, and cancellation is most complete in the regime equivalence claims occupy. Across an atlas of 1,707 paired model-by-task cells mined from public per-item evaluation dumps (1.3B-405B), churn runs roughly five times the net accuracy delta, and cells scoring identically to their baseline still disagree on individual items. In a preregistered audit of 17 equivalence claims from three registered frames (method papers, model cards, vendor documentation), 16 are eligible. None states a prospective numerical equivalence margin, and none releases task-matched per-item outputs, though 3 release outputs for other tasks only; 5 report too little to assess numerically, so a reader cannot check them at any sample size. We audit evidential sufficiency, not truth: no claim is called false. We supply the missing instrument: paired equivalence testing at a declared margin, with certification tables giving the items an evaluation needs, computed from disagreement observed under compression, not from independent-binomial variance. A controlled experiment pairs GPTQ and AWQ on byte-identical calibration samples across five seeds. Under the frozen eight-cell decision rule H3 is supported: changing the calibration draw was sufficient to reverse the observed method ordering in 5 of 8 confirmatory cells. The reporting standard we propose is five lines: declare a margin, run the paired test, report churn beside net delta, cite the sample size you met, release per-item outputs. It applies to any comparison between two models alike enough to be worth comparing. All per-item outputs, protocols and code are released.