# The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm

arXiv: https://arxiv.org/abs/2507.18553 · HTML: https://mlsanity.com/paper/2507.18553/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Builds on (most cited first, 11 of 11)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Extreme Compression of Large Language Models via Additive Quantization](https://mlsanity.com/paper/2401.06118/) (2024)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [QTIP: Quantization with Trellises and Incoherence Processing](https://mlsanity.com/paper/2406.11235/) (2024)
- [Towards the Limit of Network Quantization](https://mlsanity.com/paper/1612.01543/) (2016)
- ["Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization](https://mlsanity.com/paper/2411.02355/) (2024)
- [ICQuant: Index Coding enables Low-bit LLM Quantization](https://mlsanity.com/paper/2505.00850/) (2025)

## Built on by (most cited first, 20 of 20)

- [WUSH: Near-Optimal Adaptive Transforms for LLM Quantization](https://mlsanity.com/paper/2512.00956/) (2025)
- [The Lattice Geometry of Neural Network Quantization -- A Short Equivalence Proof of GPTQ and Babai's Algorithm](https://mlsanity.com/paper/2508.01077/) (2025)
- [WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization](https://mlsanity.com/paper/2603.04956/) (2026)
- [CoreQ: Learning-Free Mismatch Correction and Successive Rounding for Quantization](https://mlsanity.com/paper/2602.05902/) (2026)
- [High-Rate Quantized Matrix Multiplication II](https://mlsanity.com/paper/2605.13768/) (2026)
- [BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models](https://mlsanity.com/paper/2602.04163/) (2026)
- [HoloQ-VLA: Uniform W4A4 Quantization of Vision-Language-Action Models](https://mlsanity.com/paper/2605.28803/) (2026)
- [GPTQ-2D: Cubic-Time Two-Sided Adaptive Rounding](https://mlsanity.com/paper/2607.27042/) (2026)
- [SoftWater: Class-Aware Rate Allocation for Softmax Quantization](https://mlsanity.com/paper/2608.12026/) (2026)
- [Price of metric universality in vector quantization is at most 0.11 bit](https://mlsanity.com/paper/2602.05790/) (2026)
- [OJBKQ: Objective-Joint Babai-Klein Quantization](https://mlsanity.com/paper/2602.08376/) (2026)
- [GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization](https://mlsanity.com/paper/2605.26092/) (2026)
- [FastKron: Efficient Quantization with Kronecker-Factored Hessians](https://mlsanity.com/paper/2608.06291/) (2026)
- [SchurQuant: Groupwise Discrete Optimization for Layer-Wise LLM Quantization](https://mlsanity.com/paper/2608.15567/) (2026)
- [WaterKron and FlipFlop Hessian: Information-Theoretically Grounded Quantization with Kronecker-factored Hessians](https://mlsanity.com/paper/2609.14706/) (2026)
- [Generalization behavior of OPTQ and the role of regularization](https://mlsanity.com/paper/2609.31560/) (2026)
- [Product-Aware Deterministic Rounding for Quantized Matrix Multiplication](https://mlsanity.com/paper/2609.31641/) (2026)
- [EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitrates](https://mlsanity.com/paper/2609.34185/) (2026)
- [JARQ: Joint Alternating Refinement for Quantization](https://mlsanity.com/paper/2609.38599/) (2026)
- [Revisiting Error Feedback for Model Quantization](https://mlsanity.com/paper/2610.18139/) (2026)

## Abstract (arXiv)

Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators. While GPTQ emerged as one of the standard methods for one-shot post-training quantization at LLM scale, its inner workings are described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees. In this work, we show that, when executed back-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) on a lattice defined by the Hessian matrix of the layer's inputs. This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences: first, the GPTQ error propagation step gains an intuitive geometric interpretation; second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped. Leveraging this bound, we design post-training quantization methods that avoid clipping, and outperform the original GPTQ. In addition, we provide efficient GPU inference kernels for the resulting representation. Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for billion-parameter models. Source code is available at https://github.com/IST-DASLab/GPTQ-Babai.