# LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices

arXiv: https://arxiv.org/abs/2407.11534 · HTML: https://mlsanity.com/paper/2407.11534/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023)
- [QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving](https://mlsanity.com/paper/2405.04532/) (2024)
- [QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization](https://mlsanity.com/paper/2203.05740/) (2022)
- [Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss](https://mlsanity.com/paper/1808.05779/) (2018)
- [LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models](https://mlsanity.com/paper/2206.09557/) (2022)
- [Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization](https://mlsanity.com/paper/2305.14152/) (2023)
- [An empirical study of LLaMA3 quantization: from LLMs to MLLMs](https://mlsanity.com/paper/2404.14047/) (2024)
- [Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling](https://mlsanity.com/paper/2304.09145/) (2023)
- [FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization](https://mlsanity.com/paper/2306.00317/) (2023)
- [Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization](https://mlsanity.com/paper/2311.05161/) (2023)
- [Distance-aware Quantization](https://mlsanity.com/paper/2108.06983/) (2021)
- [Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss](https://mlsanity.com/paper/2109.02100/) (2021)
- [Efficient Adaptive Activation Rounding for Post-Training Quantization](https://mlsanity.com/paper/2208.11945/) (2022)

## Built on by (most cited first, 11 of 11)

- [Peri-LN: Revisiting Normalization Layer in the Transformer Architecture](https://mlsanity.com/paper/2502.02732/) (2025)
- [CoreQ: Learning-Free Mismatch Correction and Successive Rounding for Quantization](https://mlsanity.com/paper/2602.05902/) (2026)
- [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025)
- [LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs](https://mlsanity.com/paper/2605.29756/) (2026)
- [Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation](https://mlsanity.com/paper/2601.22716/) (2026)
- [Revisiting Adaptive Rounding with Vectorized Reparameterization for LLM Quantization](https://mlsanity.com/paper/2602.02151/) (2026)
- [RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference](https://mlsanity.com/paper/2603.17891/) (2026)
- [Language-Conditional Dequantization: Recovering What Quantization Steals from Non-English Languages](https://mlsanity.com/paper/2608.11786/) (2026)
- [Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation](https://mlsanity.com/paper/2608.24615/) (2026)
- [Predicting Quantization Price for Selecting PTQ Configurations Before Deployment](https://mlsanity.com/paper/2609.28270/) (2026)
- [Calibrate the Decisions That Change the Future: On-Policy Post-Training Quantization for Multimodal Large Language Models](https://mlsanity.com/paper/2609.36828/) (2026)

## Abstract (arXiv)

With the commercialization of large language models (LLMs), weight-activation quantization has emerged to compress and accelerate LLMs, achieving high throughput while reducing inference costs. However, existing post-training quantization (PTQ) techniques for quantizing weights and activations of LLMs still suffer from non-negligible accuracy drops, especially on massive multitask language understanding. To address this issue, we propose Low-Rank Quantization (LRQ) - a simple yet effective post-training weight quantization method for LLMs that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices, replacing the conventional full weight-scaling matrices that entail as many learnable scales as their associated weights. Thanks to parameter sharing via low-rank structure, LRQ only needs to learn significantly fewer parameters while enabling the individual scaling of weights, thus boosting the generalization capability of quantized LLMs. We show the superiority of LRQ over prior LLM PTQ works under (i) 8-bit weight and per-tensor activation quantization, (ii) 4-bit weight and 8-bit per-token activation quantization, and (iii) low-bit weight-only quantization schemes. Our code is available at Software.