mlsanityData to 14 Oct 2026

Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Models

arXiv 2210.15184 ↗First posted 27 Oct 2022cs.CLcited by 5

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Leveraging shared learning through Massively Multilingual Models, state-of-the-art machine translation models are often able to adapt to the paucity of data for low-resource languages. However, this performance comes at the cost of significantly bloated models which are not practically deployable. Knowledge Distillation is one popular technique to develop competitive, lightweight models: In this work, we first evaluate its use to compress MT models focusing on languages with extremely limited training data. Through our analysis across 8 languages, we find that the variance in the performance of the distilled models due to their dependence on priors including the amount of synthetic data used for distillation, the student architecture, training hyperparameters and confidence of the teacher models, makes distillation a brittle compression mechanism. To mitigate this, we explore the use of post-training quantization for the compression of these models. Here, we find that while distillation provides gains across some low-resource languages, quantization provides more consistent performance trends for the entire range of languages, especially the lowest-resource languages in our target set.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 37 of its 65 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

37 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Distilling Large Language Models into Tiny and Effective Students using pQRNN21 Jan

Other references · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2016Sequence-Level Knowledge Distillationcited by 801
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  11. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  12. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
24 more
  1. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  2. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  3. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
  4. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 128
  5. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  6. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 96
  7. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  8. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  9. 2020Vocabulary Learning via Optimal Transport for Neural Machine Translationcited by 58
  10. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  11. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  12. 2020TextHide: Tackling Data Privacy in Language Understanding Taskscited by 33
  13. 2018Marian: Cost-effective High-Quality Neural Machine Translation in C++cited by 22
  14. 2020Privacy at Scale: Introducing the PrivaSeer Corpus of Web Privacy Policiescited by 22
  15. 2022EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generationcited by 21
  16. 2021Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Studycited by 19
  17. 2022How can NLP Help Revitalize Endangered Languages? A Case Study and Roadmap for the Cherokee Languagecited by 19
  18. 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
  19. 2020Building a Multi-domain Neural Machine Translation Model using Knowledge Distillationcited by 9
  20. 2022What Do Compressed Multilingual Machine Translation Models Forget?cited by 9
  21. 2021LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillationcited by 5
  22. 2019Language Graph Distillation for Low-Resource Machine Translationcited by 4
  23. 2022Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Trainingcited by 3
  24. 2022Small Batch Sizes Improve Training of Low-Resource Neural MTcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 1.

5 papers.

  1. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  2. 2023An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Modelscited by 5
  3. 2024All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typingcited by 3
  4. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  5. 2025The Uneven Impact of Post-Training Quantization in Machine Translationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Investigating Model Compression for Neural Machine Translation in the Biomedical Domain
  2. 2023Letz Translate: Low-Resource Machine Translation for Luxembourgish
  3. 2025Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages
  4. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  5. 2026Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained Settings
  6. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees
  7. 2024Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages
  8. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models