mlsanityData to 7 Oct 2026

Low-Rank Correction for Quantized LLMs

arXiv 2412.07902 ↗First posted 10 Dec 2024stat.ML cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We consider the problem of model compression for Large Language Models (LLMs) at post-training time, where the task is to compress a well-trained model using only a small set of calibration input data. In this work, we introduce a new low-rank approach to correct for quantization errors of activations in LLMs: we propose to add low-rank weight matrices in full precision that act on the unquantized activations. We then solve a joint optimization problem over the quantized representation of the weights and additional low-rank weight matrices to quantize both weights and activations. We focus on the case of 4-bit weight-and-activation quantization (W4A4). Using ranks equivalent to 10% of the original weight matrix size, our approach reduces the accuracy gap with the original model by more than 50%. Using ranks equivalent to 30% of the original weight matrix, the accuracy gap is closed completely. We demonstrate our results on four recent LLMs, namely Llama-2, Llama-3, Phi-3 and Mixtral models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

25 of 31 references.

References · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2024Mixtral of Expertscited by 1,785
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
13 more
  1. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  2. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  3. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  4. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  5. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  6. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  7. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  8. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  9. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  10. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 58
  11. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  12. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 39
  13. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 20263BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMscited by 1
  2. 2026Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformerscited by 0