Quantization of Large Language Models with an Overdetermined Basis
Not on the map. Why?
Abstract (arXiv)
In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or tensor into two factors. The first factor maintains a small infinity norm, while the second exhibits a similarly constrained norm when multiplied by an orthogonal matrix. Surprisingly, the entries of factors after decomposition are well-concentrated around several peaks, which allows us to efficiently replace them with corresponding centroids for quantization purposes. We study the theoretical properties of the proposed approach and rigorously evaluate our compression algorithm in the context of next-word prediction tasks and on a set of downstream tasks for text classification. Our findings demonstrate that Kashin Quantization achieves competitive or superior quality in model performance while ensuring data compression, marking a significant advancement in the field of data quantization.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 15 Apr 2024.
Builds on
17 of 28 references.
References · 17
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2017Mixed Precision Trainingcited by 1,135
- 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
- 2016Training Deep Nets with Sublinear Memory Costcited by 825
- 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
- 2023The Falcon Series of Open Language Modelscited by 526
- 2021A White Paper on Neural Network Quantizationcited by 453
- 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
- 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
- 2020Bayesian Bits: Unifying Quantization and Pruningcited by 93
- 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 89
5 more
- 2019Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorizationcited by 45
- 2020Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressorcited by 45
- 2022Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reductioncited by 14
- 2022Memory-Efficient Backpropagation through Large Linear Layerscited by 4
- 2022Survey on Large Scale Neural Network Trainingcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.