mlsanityData to 7 Oct 2026

Don't Throw Away Data: Better Sequence Knowledge Distillation

arXiv 2407.10456 ↗First posted 15 Jul 2024cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

A critical component in knowledge distillation is the means of coupling the teacher and student. The predominant sequence knowledge distillation method involves supervised learning of the student against teacher-decoded outputs, and is exemplified by the current state of the art, which incorporates minimum Bayes risk (MBR) decoding. In this paper we seek to integrate MBR more tightly in distillation training, specifically by using several high scoring MBR translations, rather than a single selected sequence, thus capturing a rich diversity of teacher outputs. Our experiments on English to German and English to Japanese translation show consistent improvements over strong baseline methods for both tasks and with varying model sizes. Additionally, we conduct a detailed analysis focusing on data efficiency and capacity curse aspects to elucidate MBR-n and explore its further potential.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 16 of its 24 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

16 of 24 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  6. 2016Sequence-Level Knowledge Distillationcited by 761
  7. 2022Natural Language to Code Translation with Executioncited by 122
  8. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  9. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  10. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  11. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  12. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
4 more
  1. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
  2. 2023Faster Minimum Bayes Risk Decoding with Confidence-based Pruningcited by 34
  3. 2023MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methodscited by 33
  4. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 2.

9 papers.

  1. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  2. 2024Better Instruction-Following Through Minimum Bayes Riskcited by 19
  3. 2024Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Datacited by 10
  4. 2024Mufu: Multilingual Fused Learning for Low-Resource Translation with LLMcited by 3
  5. 2024A Bayesian Optimization Approach to Machine Translation Rerankingcited by 2
  6. 2025Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translationcited by 2
  7. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  8. 2025Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languagescited by 0
  9. 2026Consensus Group Relative Policy Optimization for Text Generationcited by 0