mlsanityData to 13 Oct 2026

Compression of Generative Pre-trained Language Models via Quantization

arXiv 2203.10705 ↗First posted 21 Mar 2022cs.CL cs.CVcited by 79

Not on the map. Why?

Abstract (arXiv)

The increasing size of generative Pre-trained Language Models (PLMs) has greatly increased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to compress generative PLMs, and the underlying difficulty remains unclear. In this paper, we compress generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the textithomogeneous word embeddings caused by reduced capacity, and textitvaried distribution of weights. Correspondingly, we propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules. Empirical results on various tasks show that our proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. With comparable performance with the full-precision models, we achieve 14.4x and 13.4x compression rates on GPT-2 and BART, respectively.

Key facts. Cited by 79 papers in the map (data through 13 Oct 2026). Builds on 37 of its 42 references that are in the map. First posted on arXiv 21 Mar 2022.

Builds on

37 of 42 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,730
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,472
  12. 2016Pointer Sentinel Mixture Modelscited by 3,047
25 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,860
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,056
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,312
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,157
  6. 2020On Layer Normalization in the Transformer Architecturecited by 910
  7. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 824
  8. 2019Contrastive Representation Distillationcited by 739
  9. 2019Learned Step Size Quantizationcited by 621
  10. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 589
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 480
  12. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 465
  13. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  14. 2019Q8BERT: Quantized 8Bit BERTcited by 299
  15. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 247
  16. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 233
  17. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 177
  18. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 147
  19. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  20. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  21. 2016Loss-aware Binarization of Deep Networkscited by 56
  22. 2018Loss-aware Weight Quantization of Deep Networkscited by 31
  23. 2021Kronecker Decomposition for GPT Compressioncited by 27
  24. 2020LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuningcited by 25
  25. 2022SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Trainingcited by 19

Built on by

Cited by year

Citing papers per year: 2022: 13; 2023: 28; 2024: 31; 2025: 7.

79 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,559
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
  3. 2023A Comprehensive Overview of Large Language Modelscited by 644
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 522
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 279
  6. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 231
  7. 2023Language Modeling Is Compressioncited by 225
  8. 2023Efficient Large Language Models: A Surveycited by 185
  9. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 179
  10. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  11. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 116
  12. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
38 more
  1. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  3. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 64
  4. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  5. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 52
  6. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  7. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 44
  8. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  9. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  10. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 31
  11. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  12. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 24
  13. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  14. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 21
  15. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 21
  16. 2023Token-Scaled Logit Distillation for Ternary Weight Generative Language Modelscited by 20
  17. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  18. 2022SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languagescited by 16
  19. 2024Squat: Quant Small Language Models on the Edgecited by 16
  20. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  21. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  22. 2024Synergy-of-Thoughts: Eliciting Efficient Reasoning in Hybrid Language Modelscited by 13
  23. 2024Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Datacited by 13
  24. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
  25. 2022Quadapter: Adapter for GPT-2 Quantizationcited by 10
  26. 2024RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterizationcited by 10
  27. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  28. 2025The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exitcited by 10
  29. 2022What Do Compressed Multilingual Machine Translation Models Forget?cited by 9
  30. 2022Differentiable Data Augmentation for Contrastive Sentence Representation Learningcited by 9
  31. 2023FedYolo: Augmenting Federated Learning with Pretrained Transformerscited by 9
  32. 2023The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Modelscited by 9
  33. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantizationcited by 9
  34. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8
  35. 2023Binary and Ternary Natural Language Generationcited by 7
  36. 2023Dynamic Stashing Quantization for Efficient Transformer Trainingcited by 6
  37. 2024From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Modelscited by 6
  38. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Modelscited by 5

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization
  2. 2022Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  4. 2023A Survey on Model Compression for Large Language Models
  5. 2021Prune Once for All: Sparse Pre-Trained Language Models
  6. 2023SqueezeLLM: Dense-and-Sparse Quantization
  7. 2026Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization
  8. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficient