mlsanityData to 14 Oct 2026

Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages

arXiv 2507.21568 ↗First posted 29 Jul 2025cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

This paper explores sequence-level knowledge distillation (KD) of multilingual pre-trained encoder-decoder translation models. We argue that the teacher model's output distribution holds valuable insights for the student, beyond the approximated mode obtained through beam search (the standard decoding method), and present Multi-Hypothesis Distillation (MHD), a sequence-level KD method that generates multiple translations for each source sentence. This provides a larger representation of the teacher model distribution and exposes the student model to a wider range of target-side prefixes. We leverage n-best lists from beam search to guide the student's learning and examine alternative decoding methods to address issues like low variability and the under-representation of infrequent tokens. For low-resource languages, our research shows that while sampling methods may slightly compromise translation quality compared to beam search based approaches, they enhance the generated corpora with greater variability and lexical richness. This ultimately improves student model performance and mitigates the gender bias amplification often associated with KD.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 50 of its 73 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

50 of 73 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Self-Evolution Knowledge Distillation for LLM-based Machine Translation19 Dec
  2. 2024Don't Throw Away Data: Better Sequence Knowledge Distillation15 Jul
  3. 2024MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation14 Mar
  4. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs19 Feb
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes23 Jun
  6. 2023Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation14 May
  7. 2023An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models19 Apr
  8. 2023Letz Translate: Low-Resource Machine Translation for Luxembourgish2 Mar

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2018Hierarchical Neural Story Generationcited by 1,193
  9. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  10. 2016Sequence-Level Knowledge Distillationcited by 801
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  12. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
30 more
  1. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  2. 2019Evaluating Gender Bias in Machine Translationcited by 249
  3. 2022A Contrastive Framework for Neural Text Generationcited by 225
  4. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  5. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  6. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 138
  7. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  8. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114
  9. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  10. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 96
  11. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  12. 2022On Decoding Strategies for Neural Text Generatorscited by 90
  13. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  14. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 65
  15. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 61
  16. 2024From LLM to NMT: Advancing Low-Resource Machine Translation with Claudecited by 58
  17. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  18. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  19. 2018Importance of Search and Evaluation Strategies in Neural Dialogue Modelingcited by 44
  20. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  21. 2023MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methodscited by 33
  22. 2023AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languagescited by 24
  23. 2024Linear-time Minimum Bayes Risk Decoding with Reference Aggregationcited by 21
  24. 2019Generalizing Back-Translation in Neural Machine Translationcited by 18
  25. 2020Decoding Methods for Neural Narrative Generationcited by 18
  26. 2024Mitigating Metric Bias in Minimum Bayes Risk Decodingcited by 16
  27. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  28. 2025SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?cited by 10
  29. 2022Exploring Diversity in Back Translation for Low-Resource Machine Translationcited by 5
  30. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Unraveling Key Factors of Knowledge Distillation
  2. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  3. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Models
  4. 2026Evolving Knowledge Distillation for Lightweight Neural Machine Translation
  5. 2021Multilingual Neural Machine Translation:Can Linguistic Hierarchies Help?
  6. 2024Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages
  7. 2021Selective Knowledge Distillation for Neural Machine Translation
  8. 2026Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models