mlsanityData to 7 Oct 2026

Towards Efficient Post-training Quantization of Pre-trained Language Models

arXiv 2109.15082 ↗First posted 30 Sep 2021cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Network quantization has gained increasing attention with the rapid growth of large pre-trained language models~(PLMs). However, most existing quantization methods for PLMs follow quantization-aware training~(QAT) that requires end-to-end training with full access to the entire dataset. Therefore, they suffer from slow training, large memory overhead, and data security issues. In this paper, we study post-training quantization~(PTQ) of PLMs, and propose module-wise quantization error minimization~(MREM), an efficient solution to mitigate these issues. By partitioning the PLM into multiple modules, we minimize the reconstruction error incurred by quantization for each module. In addition, we design a new model parallel training strategy such that each module can be trained locally on separate computing devices without waiting for preceding modules, which brings nearly the theoretical training speed-up (e.g., 4× on 4 GPUs). Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 59 of its 98 references that are in the map. First posted on arXiv 30 Sep 2021.

Builds on

59 of 98 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
38 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2019Are Sixteen Heads Really Better than One?cited by 803
  5. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  6. 2018Universal Transformerscited by 613
  7. 2019Learned Step Size Quantizationcited by 604
  8. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  9. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  11. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  13. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  14. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  15. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  16. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  17. 2021Post-Training Quantization for Vision Transformercited by 274
  18. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  19. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  20. 2016Ternary Weight Networkscited by 195
  21. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  22. 2019The Evolved Transformercited by 180
  23. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  24. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  25. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  26. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  27. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  28. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  29. 2019Loss Aware Post-training Quantizationcited by 94
  30. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 89
  31. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
  32. 2016Loss-aware Binarization of Deep Networkscited by 56
  33. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
  34. 2020AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Searchcited by 51
  35. 2020Discrimination-aware Network Pruning for Deep Model Compressioncited by 50
  36. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  37. 2020Distilling Knowledge by Mimicking Featurescited by 34
  38. 2018Loss-aware Weight Quantization of Deep Networkscited by 31

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 16; 2025: 4; 2026: 2.

37 papers.

Sort:

By area:

  1. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  2. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  3. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  4. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  5. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  6. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  7. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  8. 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
  9. 2023FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantizationcited by 46
  10. 2024A Survey on Transformer Compressioncited by 44
  11. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  12. 2023Efficient Quantization Strategies for Latent Diffusion Modelscited by 16
25 more
  1. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  2. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
  3. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  4. 2025The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exitcited by 10
  5. 2024FAIR Enough: How Can We Develop and Assess a FAIR-Compliant Dataset for Large Language Models' Training?cited by 9
  6. 2023Binary and Ternary Natural Language Generationcited by 7
  7. 2022Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementationcited by 6
  8. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inferencecited by 6
  9. 2025Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performancecited by 4
  10. 2024An Analysis on Quantizing Diffusion Transformerscited by 3
  11. 2023Augmenting Hessians with Inter-Layer Dependencies for Mixed-Precision Post-Training Quantizationcited by 2
  12. 2023Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Modelscited by 2
  13. 2025Low-Rank Compression of Language Models via Differentiable Rank Selectioncited by 2
  14. 2023Exploring Post-Training Quantization of Protein Language Modelscited by 1
  15. 2023Herd: Using multiple, smaller LLMs to match the performances of proprietary, large LLMs via an intelligent composercited by 1
  16. 2024LCQ: Low-Rank Codebook based Quantization for Large Language Modelscited by 1
  17. 2024Privacy-Aware Randomized Quantization via Linear Programmingcited by 1
  18. 2024Predicting Probabilities of Error to Combine Quantization and Early Exiting: QuEEcited by 1
  19. 2024A Bag of Tricks for Scaling CPU-based Deep FFMs to more than 300m Predictions per Secondcited by 1
  20. 2026Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptationcited by 1
  21. 2023Rediscovering Hashed Random Projections for Efficient Quantization of Contextualized Sentence Embeddingscited by 0
  22. 2024Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Datacited by 0
  23. 2024What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Modelscited by 0
  24. 2024Artificial Neural Networks for Photonic Applications: From Algorithms to Implementationcited by 0
  25. 2026QuaSAR: Quantization Compensation via Stable Activation-Aware Rank Truncationcited by 0