mlsanityData to 14 Oct 2026

The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference

arXiv 2608.13756 ↗First posted 13 Aug 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable. We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel (CUTLASS versus Triton) inside vLLM. At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64). What makes this more than a benchmark discrepancy is an integer alibi: for shared INT8 operands under a verified no-overflow bound, the INT32 dot product is exact and order-independent, so the accumulator cannot be the source of any difference. Feeding both kernels identical operands from every linear layer of Qwen3-1.7B and 8B (196 and 252 layers), we find bit-identical outputs under power-of-two scales, confirming a pinned prediction list 196/196 and 252/252 (pre-registered at 1.7B, pinned but not blind at 8B), and observed differences of at most one bfloat16 spacing under the checkpoints' real scales. This localizes the divergence to scale application and output rounding after the exact accumulator. Applied as a probe checkpoint, the same intervention restores end-to-end bitwise agreement (8/8 and 16/16 sequences). Cross-implementation FP8 GEMM shows a different signature: both the prevalence and the magnitude of differences grow with reduction depth, while the INT8 fraction stays at parts per million and within one spacing over a 64x range of K. Teacher-forced replay ties layers to tokens: flips concentrate at small logit margins, which predict flip risk with ROC-AUC 0.94 on 16,384 positions. We will release the pre-registration, per-layer predictions, manifests with kernel-selection evidence, and a conformance procedure that turns these controls into a concrete check for kernel interchangeability.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

28 of 50 references.

References · 28

  1. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  2. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  3. 2023Microscaling Data Formats for Deep Learningcited by 152
  4. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 53
  5. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  6. 2024Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applicationscited by 24
  7. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 16
  8. 2022RAPQ: Rescuing Accuracy for Power-of-Two Low-bit Post-training Quantizationcited by 15
  9. 2025Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatchcited by 14
  10. 2023Pre-registration for Predictive Modelingcited by 13
  11. 2023A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidancecited by 12
  12. 2025LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Servingcited by 8
16 more
  1. 2026The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibilitycited by 8
  2. 2025Equivalence Checking of ML GPU Kernelscited by 7
  3. 2025An SMT Formalization of Mixed-Precision Matrix Multiplication: Modeling Three Generations of Tensor Corescited by 5
  4. 2023DGEMM on Integer Matrix Multiplication Unitcited by 4
  5. 2026Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminismcited by 4
  6. 2026Bit-Exact AI Inference Verification Without Performance Tradeoffscited by 4
  7. 2026Kernel Contracts: A Specification Language for ML Kernel Correctness Across Heterogeneous Siliconcited by 3
  8. 2026MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inferencecited by 3
  9. 2025DiFR: Inference Verification Despite Nondeterminismcited by 2
  10. 2026Hawkeye: Reproducing GPU-Level Non-Determinismcited by 2
  11. 2026Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEALcited by 2
  12. 2026Fingerprinting Inference Systems of Large Language Modelscited by 1
  13. 2026Training-Inference Kernel Contracts: Bounding Divergence in Post-Training and Deploymentcited by 1
  14. 2026Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUscited by 1
  15. 2026Which Decisions Low-Bit Quantization Breaks, and How to Predict Themcited by 1
  16. 2026Spec Sheets Are Not Kernels: An ISA- and Source-Level Audit of INT8 Availability on NVIDIA Blackwell Ultracited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformancecited by 1
  2. 2026Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Acceleratorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models
  2. 2026GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models
  3. 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
  4. 2026TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs
  5. 2025VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference
  6. 2026PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
  7. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  8. 2026Dissecting GPU Utilization for LLM Inference on Nvidia Hopper