mlsanityData to 14 Oct 2026

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

arXiv 2206.01861 ↗First posted 4 Jun 2022cs.CL cs.LGcited by 526

Not on the map. Why?

Cited by 62 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

How to efficiently serve ever-larger trained natural language models in practice has become exceptionally challenging even for powerful cloud servers due to their prohibitive memory/computation requirements. In this work, we present an efficient and affordable post-training quantization approach to compress large Transformer-based models, termed as ZeroQuant. ZeroQuant is an end-to-end quantization and inference pipeline with three main components: (1) a fine-grained hardware-friendly quantization scheme for both weight and activations; (2) a novel affordable layer-by-layer knowledge distillation algorithm (LKD) even without the access to the original training data; (3) a highly-optimized quantization system backend support to remove the quantization/dequantization overhead. As such, we are able to show that: (1) ZeroQuant can reduce the precision for weights and activations to INT8 in a cost-free way for both BERT and GPT3-style models with minimal accuracy impact, which leads to up to 5.19x/4.16x speedup on those models compared to FP16 inference; (2) ZeroQuant plus LKD affordably quantize the weights in the fully-connected module to INT4 along with INT8 weights in the attention module and INT8 activations, resulting in 3x memory footprint reduction compared to the FP16 model; (3) ZeroQuant can be directly applied to two of the largest open-sourced language models, including GPT-J6B and GPT-NeoX20, for which our INT8 model achieves similar accuracy as the FP16 model but achieves up to 5.2x better efficiency.

Key facts. Cited by 526 papers in the map (data through 14 Oct 2026). Builds on 60 of its 85 references that are in the map. First posted on arXiv 4 Jun 2022.

Builds on

60 of 85 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  13. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  14. 2018Neural Network Acceptability Judgmentscited by 958
  15. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  16. 2019Are Sixteen Heads Really Better than One?cited by 816
  17. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  18. 2019Learned Step Size Quantizationcited by 628
  19. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  20. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  21. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  22. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  23. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  24. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  25. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  26. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  27. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  28. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  29. 2021Post-Training Quantization for Vision Transformercited by 278
  30. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  31. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  32. 2016Ternary Weight Networkscited by 204
  33. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  34. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  35. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  36. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  37. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  38. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 109; 2024: 166; 2025: 140; 2026: 102.

526 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  9. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
38 more
  1. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  2. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  3. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  4. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  5. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  6. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  7. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  8. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  9. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  10. 2023A Survey on Model Compression for Large Language Modelscited by 269
  11. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  12. 2023Challenges and Applications of Large Language Modelscited by 254
  13. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  14. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  15. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  16. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  17. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  18. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  19. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  20. 2023Efficient Large Language Models: A Surveycited by 186
  21. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  22. 2023Speculative Decoding with Big Little Decodercited by 164
  23. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  24. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  25. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  26. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  27. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  28. 2023REST: Retrieval-Based Speculative Decodingcited by 150
  29. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  30. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  31. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  32. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  33. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 115
  34. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  35. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  36. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  37. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 97
  38. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93

50 most cited of 526 All 526 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficient
  2. 2022Zero-Shot Dynamic Quantization for Transformer Inference
  3. 2025FPTQuant: Function-Preserving Transforms for LLM Quantization
  4. 2025GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
  5. 2024SpinQuant: LLM quantization with learned rotations
  6. 2023Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers
  7. 2026RUQuant: Towards Refining Uniform Quantization for Large Language Models
  8. 2023Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models