mlsanityData to 14 Oct 2026

Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs

arXiv 2410.14570 ↗First posted 18 Oct 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models of high parameter counts are computationally expensive, yet can be made much more efficient by compressing their weights to very low numerical precision. This can be achieved either through post-training quantization by minimizing local, layer-wise quantization errors, or through quantization-aware fine-tuning by minimizing the global loss function. In this study, we discovered that, under the same data constraint, the former approach nearly always fared worse than the latter, a phenomenon particularly prominent when the numerical precision is very low. We further showed that this difficulty of post-training quantization arose from stark misalignment between optimization of the local and global objective functions. Our findings explains limited utility in minimization of local quantization error and the importance of direct quantization-aware fine-tuning, in the regime of large models at very low precision.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

39 of 51 references.

References · 39

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
27 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  2. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  4. 2021GPT Understands, Toocited by 716
  5. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  6. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  7. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  8. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  9. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  10. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  12. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  13. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  14. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  15. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  16. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  17. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  18. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  19. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  20. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  21. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  22. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  23. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  24. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  25. 2023Building Emotional Support Chatbots in the Era of LLMscited by 62
  26. 2024Weaver: Foundation Models for Creative Writingcited by 27
  27. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Highly Efficient and Effective LLMs with Multi-Boolean Architecturescited by 2
  2. 2025The Impact of Quantization on Large Reasoning Model Reinforcement Learningcited by 2
  3. 2026MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization
  2. 2024Scaling Laws for Post Training Quantized Large Language Models
  3. 2025Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models
  4. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
  5. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?
  6. 2026SliderQuant: Accurate Post-Training Quantization for LLMs
  7. 2024Post Training Quantization of Large Language Models with Microscaling Formats
  8. 2026Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization