Contraction-Gauge Preconditioning for Quantized Matrix Multiplication
Not on the map. Why?
Abstract (arXiv)
We study low-precision computation of C=AB with both factors quantized. We derive an exact finite-dimensional identity for the expected squared product error under independent, zero-mean entrywise errors with known variance fields; it holds exactly for non-overloading subtractive dither and for independent stochastic rounding, and we empirically assess deterministic round-to-nearest (RTN). Using the product-preserving equivalence AB=(AT)(T^{-1}B), we formulate contraction-gauge preconditioning: jointly choosing a factor representation and its sharing pattern before quantization. Preconditioning can reduce product error but may require extra transformed, quantized copies of the opposite operand: a shared transform needs one copy, a block-specific transform up to one per block. Within the bounded family of positive diagonal gauges (folds), a geometric program computes a globally optimal shared fold and a linear program decides whether the identity fold is already optimal. For other families we derive computable selection statistics -- tail index for scaling, profile spread for partitioning, coherence and weighted-Gram energy for rotations, slice-energy covariance for hierarchy depth -- with upper bounds for ranking heuristic candidates. Across twelve linear products from a trained three-block image classifier, median within-product rank correlations between dither-model predictions and deterministic-RTN errors are 0.937 at 8 bits and 0.918 at 4 bits. The GP fold cuts held-out product error over the identity fold by 18.0% (8-bit) and 20.5% (4-bit) in geometric mean, beats a SmoothQuant-style grid baseline at both precisions and on ten of twelve products, and lowers composed logit MSE by 15.4% and 26.4%. We thus provide exact stochastic product-error accounting, certified selection within the diagonal family, and a common objective for evaluating reusable transform candidates under RTN.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 58 references that are in the map. First posted on arXiv 21 Jul 2026.
Builds on
31 of 58 references.
References · 31
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
- 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
- 2024SpinQuant: LLM quantization with learned rotationscited by 446
- 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
- 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
- 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
- 2016Distributed Mean Estimation with Limited Communicationcited by 134
19 more
- 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
- 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
- 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
- 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
- 2022FP8 Quantization: The Power of the Exponentcited by 87
- 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 64
- 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 53
- 2019Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorizationcited by 45
- 2024MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantizationcited by 26
- 2025NestQuant: Nested Lattice Quantization for Matrix Products and LLMscited by 16
- 2024Optimal Quantization for Matrix Multiplicationcited by 10
- 2026Pushing the Limits of Block Rotations in Post-Training Quantizationcited by 9
- 2013Gauge optimization and dualitycited by 6
- 2026High-Rate Quantized Matrix Multiplication Icited by 6
- 2026Provable Quantization with Randomized Hadamard Transformcited by 6
- 2026Dissecting Quantization Error: A Concentration-Alignment Perspectivecited by 3
- 2024Foundations of Large Language Model Compression -- Part 1: Weight Quantizationcited by 2
- 2026High-Rate Quantized Matrix Multiplication IIcited by 2
- 2026Optimal Scalar Quantization for Matrix Multiplication: Closed-Form Density and Phase Transitioncited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.