mlsanityData to 14 Oct 2026

Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models

arXiv 2307.05972 ↗First posted 12 Jul 2023cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models. We present a new method called self-distilled quantization (SDQ) that minimizes accumulative quantization errors and outperforms baselines. We apply SDQ to multilingual models XLM-R-Base and InfoXLM-Base and demonstrate that both models can be reduced from 32-bit floating point weights to 8-bit integer weights while maintaining a high level of performance on the XGLUE benchmark. Our results also highlight the challenges of quantizing multilingual models, which must generalize to languages they were not fine-tuned on.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

28 of 39 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  5. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  7. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  8. 2018Model compression via distillation and quantizationcited by 289
  9. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  10. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  11. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  12. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
16 more
  1. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  2. 2020Characterising Bias in Compressed Modelscited by 167
  3. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  4. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 151
  5. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  6. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  7. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  8. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  9. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101
  10. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
  11. 2020An Overview of Neural Network Compressioncited by 59
  12. 2020Adaptive Gradient Quantization for Data-Parallel SGDcited by 45
  13. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
  14. 2019Fully Quantized Transformer for Machine Translationcited by 36
  15. 2020One Weight Bitwidth to Rule Them Allcited by 13
  16. 2020Position-based Scaled Gradient for Model Quantization and Pruningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  2. 2026OffQ: Taming Structured Outliers in LLM Quantization by Offsettingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring Quantization for Efficient Pre-Training of Transformer Language Models
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  3. 2023Dynamic Stashing Quantization for Efficient Transformer Training
  4. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Size
  5. 2026TernaryLM: Memory-Efficient Language Modeling via Native 1.5-Bit Quantization with Adaptive Layer-wise Scaling
  6. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  7. 2025SiLQ: Simple Large Language Model Quantization-Aware Training
  8. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit