# The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference

arXiv: https://arxiv.org/abs/2608.13756 · HTML: https://mlsanity.com/paper/2608.13756/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 28 of 28)

- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](https://mlsanity.com/paper/2506.09501/) (2025)
- [FP8 versus INT8 for efficient deep learning inference](https://mlsanity.com/paper/2303.17951/) (2023)
- [Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications](https://mlsanity.com/paper/2408.05148/) (2024)
- [LLM-42: Enabling Determinism in LLM Inference with Verified Speculation](https://mlsanity.com/paper/2601.17768/) (2026)
- [RAPQ: Rescuing Accuracy for Power-of-Two Low-bit Post-training Quantization](https://mlsanity.com/paper/2204.12322/) (2022)
- [Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch](https://mlsanity.com/paper/2511.17826/) (2025)
- [Pre-registration for Predictive Modeling](https://mlsanity.com/paper/2311.18807/) (2023)
- [A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidance](https://mlsanity.com/paper/2308.13504/) (2023)
- [LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving](https://mlsanity.com/paper/2509.01229/) (2025)
- [The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility](https://mlsanity.com/paper/2605.19537/) (2026)
- [Equivalence Checking of ML GPU Kernels](https://mlsanity.com/paper/2511.12638/) (2025)
- [An SMT Formalization of Mixed-Precision Matrix Multiplication: Modeling Three Generations of Tensor Cores](https://mlsanity.com/paper/2502.15999/) (2025)
- [DGEMM on Integer Matrix Multiplication Unit](https://mlsanity.com/paper/2306.11975/) (2023)
- [Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism](https://mlsanity.com/paper/2601.06118/) (2026)
- [Bit-Exact AI Inference Verification Without Performance Tradeoffs](https://mlsanity.com/paper/2606.00279/) (2026)
- [Kernel Contracts: A Specification Language for ML Kernel Correctness Across Heterogeneous Silicon](https://mlsanity.com/paper/2604.22032/) (2026)
- [MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference](https://mlsanity.com/paper/2605.30218/) (2026)
- [DiFR: Inference Verification Despite Nondeterminism](https://mlsanity.com/paper/2511.20621/) (2025)
- [Hawkeye: Reproducing GPU-Level Non-Determinism](https://mlsanity.com/paper/2603.20421/) (2026)
- [Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL](https://mlsanity.com/paper/2606.21023/) (2026)
- [Fingerprinting Inference Systems of Large Language Models](https://mlsanity.com/paper/2605.29979/) (2026)
- [Training-Inference Kernel Contracts: Bounding Divergence in Post-Training and Deployment](https://mlsanity.com/paper/2606.07581/) (2026)
- [Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs](https://mlsanity.com/paper/2607.11368/) (2026)
- [Which Decisions Low-Bit Quantization Breaks, and How to Predict Them](https://mlsanity.com/paper/2608.06564/) (2026)
- [Spec Sheets Are Not Kernels: An ISA- and Source-Level Audit of INT8 Availability on NVIDIA Blackwell Ultra](https://mlsanity.com/paper/2608.11693/) (2026)

## Built on by (most cited first, 2 of 2)

- [Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance](https://mlsanity.com/paper/2609.00363/) (2026)
- [Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators](https://mlsanity.com/paper/2609.16085/) (2026)

## Abstract (arXiv)

Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable. We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel (CUTLASS versus Triton) inside vLLM. At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64). What makes this more than a benchmark discrepancy is an integer alibi: for shared INT8 operands under a verified no-overflow bound, the INT32 dot product is exact and order-independent, so the accumulator cannot be the source of any difference. Feeding both kernels identical operands from every linear layer of Qwen3-1.7B and 8B (196 and 252 layers), we find bit-identical outputs under power-of-two scales, confirming a pinned prediction list 196/196 and 252/252 (pre-registered at 1.7B, pinned but not blind at 8B), and observed differences of at most one bfloat16 spacing under the checkpoints' real scales. This localizes the divergence to scale application and output rounding after the exact accumulator. Applied as a probe checkpoint, the same intervention restores end-to-end bitwise agreement (8/8 and 16/16 sequences). Cross-implementation FP8 GEMM shows a different signature: both the prevalence and the magnitude of differences grow with reduction depth, while the INT8 fraction stays at parts per million and within one spacing over a 64x range of K. Teacher-forced replay ties layers to tokens: flips concentrate at small logit margins, which predict flip risk with ROC-AUC 0.94 on 16,384 positions. We will release the pre-registration, per-layer predictions, manifests with kernel-selection evidence, and a conformance procedure that turns these controls into a concrete check for kernel interchangeability.