mlsanityData to 14 Oct 2026

Extreme Compression for Pre-trained Transformers Made Simple and Efficient

arXiv 2206.01859 ↗First posted 4 Jun 2022cs.CL cs.LGcited by 19

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Extreme compression, particularly ultra-low bit precision (binary/ternary) quantization, has been proposed to fit large NLP models on resource-constraint devices. However, to preserve the accuracy for such aggressive compression schemes, cutting-edge methods usually introduce complicated compression pipelines, e.g., multi-stage expensive knowledge distillation with extensive hyperparameter tuning. Also, they oftentimes focus less on smaller transformer models that have already been heavily compressed via knowledge distillation and lack a systematic study to show the effectiveness of their methods. In this paper, we perform a very comprehensive systematic study to measure the impact of many key hyperparameters and training strategies from previous works. As a result, we find out that previous baselines for ultra-low bit precision quantization are significantly under-trained. Based on our study, we propose a simple yet effective compression pipeline for extreme compression, named XTC. XTC demonstrates that (1) we can skip the pre-training knowledge distillation to obtain a 5-layer BERT while achieving better performance than previous state-of-the-art methods, e.g., the 6-layer TinyBERT; (2) extreme quantization plus layer reduction is able to reduce the model size by 50x, resulting in new state-of-the-art results on GLUE tasks.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 52 of its 68 references that are in the map. First posted on arXiv 4 Jun 2022.

Builds on

52 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing31 May

Other references · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2016Layer Normalizationcited by 5,669
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
38 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  7. 2018Neural Network Acceptability Judgmentscited by 958
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  9. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  10. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  13. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  14. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  15. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  16. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  17. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  18. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  19. 2021Post-Training Quantization for Vision Transformercited by 278
  20. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  21. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  22. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  23. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  24. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  25. 2016Ternary Weight Networkscited by 204
  26. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  27. 2021Block Pruning For Faster Transformerscited by 177
  28. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  29. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  30. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  31. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  32. 2019Knowledge Distillation from Internal Representationscited by 87
  33. 2022BiBERT: Accurate Fully Binarized BERTcited by 73
  34. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  35. 2016Local Binary Convolutional Neural Networkscited by 38
  36. 2021AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Modelscited by 31
  37. 2016Bitwise Neural Networkscited by 29
  38. 2017Ternary Neural Networks with Fine-Grained Quantizationcited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 10; 2024: 6; 2025: 1.

19 papers.

Sort:

By area:

  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2023Speculative Decoding with Big Little Decodercited by 164
  3. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  4. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  5. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  6. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  7. 2024A Comprehensive Study on Quantization Techniques for Large Language Modelscited by 36
  8. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  9. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  10. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbationcited by 23
  11. 2023Token-Scaled Logit Distillation for Ternary Weight Generative Language Modelscited by 20
  12. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
7 more
  1. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  2. 2025SpectR: Dynamically Composing LM Experts with Spectral Routingcited by 6
  3. 2023PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Modelscited by 5
  4. 2024A Transformer-Based Approach for Smart Invocation of Automatic Code Completioncited by 5
  5. 2024Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsitycited by 5
  6. 2023Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformerscited by 2
  7. 2024Athena: Efficient Block-Wise Post-Training Quantization for Large Language Models Using Second-Order Matrix Derivative Informationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2022Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding
  2. 2022Exploring Extreme Parameter Compression for Pre-trained Language Models
  3. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  4. 2021Prune Once for All: Sparse Pre-Trained Language Models
  5. 2023Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding
  6. 2022Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks
  7. 2023Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models
  8. 2022Compression of Generative Pre-trained Language Models via Quantization