mlsanityData to 7 Oct 2026

ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

arXiv 2303.08302 ↗First posted 15 Mar 2023cs.LG cs.AI cs.CLcited by 78

Not on the map. Why?

Abstract (arXiv)

Post-training quantization (PTQ) has emerged as a promising technique for mitigating memory consumption and computational costs in large language models (LLMs). However, a systematic examination of various quantization schemes, model families, and quantization bit precision has been absent from the literature. In this paper, we conduct a comprehensive analysis of these factors by investigating the effects of PTQ on weight-only, activation-only, and weight-and-activation quantization using diverse methods such as round-to-nearest (RTN), GPTQ, ZeroQuant, and their variants. We apply these methods to two distinct model families with parameters ranging from 125M to 176B. Our contributions include: (1) a sensitivity analysis revealing that activation quantization is generally more susceptible to weight quantization, with smaller models often outperforming larger models in terms of activation quantization; (2) an evaluation and comparison of existing PTQ methods to optimize model size reduction while minimizing the impact on accuracy, revealing that none of the current methods can achieve the original model quality for quantization with either INT4-weight or INT4-weight-and-INT8-activation; (3) based on these insights, we propose an optimized method called Low-Rank Compensation (LoRC), which employs low-rank matrices to enhance model quality recovery with a minimal increase in model size.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

31 of 41 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
19 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2019Learned Step Size Quantizationcited by 604
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  7. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  8. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  9. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  10. 2018Model compression via distillation and quantizationcited by 287
  11. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  12. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  13. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  14. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  15. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  16. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  17. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  18. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  19. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 32; 2025: 14; 2026: 15.

78 papers.

Sort:

By area:

  1. 2023A Survey on Model Compression for Large Language Modelscited by 263
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2023Efficient Large Language Models: A Surveycited by 183
  5. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  6. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  7. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  8. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  9. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  10. 2024Evaluating Quantized Large Language Modelscited by 75
  11. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  12. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 62
38 more
  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  2. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 58
  3. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 49
  4. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  5. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  6. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  7. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  8. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33
  9. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  10. 2023QuantEase: Optimization-based Quantization for Language Modelscited by 28
  11. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  12. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbationcited by 22
  13. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 21
  14. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 20
  15. 2023FPTQ: Fine-grained Post-Training Quantization for Large Language Modelscited by 16
  16. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  17. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  18. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  19. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 12
  20. 2024FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Designcited by 11
  21. 2023A Speed Odyssey for Deployable Quantization of LLMscited by 10
  22. 2024Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behindcited by 10
  23. 2024Collage: Light-Weight Low-Precision Strategy for LLM Trainingcited by 9
  24. 2024Inference Optimization of Foundation Models on AI Acceleratorscited by 9
  25. 2024RILQ: Rank-Insensitive LoRA-based Quantization Error Compensation for Boosting 2-bit Large Language Model Accuracycited by 9
  26. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniquescited by 9
  27. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  28. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inferencecited by 6
  29. 2024Dynamic Sentiment Analysis with Local Large Language Models using Majority Voting: A Study on Factors Affecting Restaurant Evaluationcited by 5
  30. 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initializationcited by 5
  31. 2026MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelscited by 5
  32. 2023Performance of Large Language Models in a Computer Science Degree Programcited by 4
  33. 2024Dynamic Q&A of Clinical Documents with Large Language Modelscited by 4
  34. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  35. 20254bit-Quantization in Vector-Embedding for RAGcited by 4
  36. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Cachingcited by 3
  37. 2025Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Modelscited by 3
  38. 2026MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantizationcited by 3

50 most cited of 78 All 78 (JSON)