KD4MT: A Survey of Knowledge Distillation for Machine Translation
Not on the map. Why?
Abstract (arXiv)
Knowledge Distillation (KD) as a research area has gained a lot of traction in recent years as a compression tool to address challenges related to ever-larger models in NLP. Remarkably, Machine Translation (MT) offers a much more nuanced take on this narrative: in MT, KD also functions as a general-purpose knowledge transfer mechanism that shapes supervision and translation quality as well as efficiency. This survey synthesizes KD for MT (KD4MT) across 105 papers (through October 1, 2025). We begin by introducing both MT and KD for non-experts, followed by an overview of the standard KD approaches relevant to MT applications. Subsequently, we categorize advances in the KD4MT literature based on (i) their methodological contributions and (ii) their practical applications. Our qualitative and quantitative analyses identify common trends in the field and highlight key research gaps as well as the absence of unified evaluation practice for KD methods in MT. We further provide practical guidelines for selecting a KD method in concrete settings and highlight potential risks associated with the application of KD to MT such as increased hallucination and bias amplification. Finally, we discuss the role of LLMs in re-shaping the KD4MT field. To support further research, we complement our survey with a publicly available database summarizing the main characteristics of the surveyed KD methods and a glossary of key terms.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 85 of its 148 references that are in the map. First posted on arXiv 22 Jan 2026.
Builds on
85 of 148 references.
References · 85
- 2017Attention Is All You Needcited by 59,932
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2020Knowledge Distillation: A Surveycited by 1,578
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
- 2016Sequence-Level Knowledge Distillationcited by 761
- 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
- 2017Non-Autoregressive Neural Machine Translationcited by 457
- 2017Six Challenges for Neural Machine Translationcited by 334
- 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
38 more
- 2019Massively Multilingual Neural Machine Translationcited by 246
- 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
- 2019Levenshtein Transformercited by 197
- 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
- 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
- 2023Hallucinations in Large Multilingual Translation Modelscited by 126
- 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
- 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
- 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
- 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
- 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
- 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
- 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
- 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
- 2020Feature-map-level Online Adversarial Knowledge Distillationcited by 62
- 2021Annealing Knowledge Distillationcited by 61
- 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
- 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 55
- 2018Trivial Transfer Learning for Low-Resource Neural Machine Translationcited by 47
- 2019Multilingual Neural Machine Translation with Language Clusteringcited by 47
- 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
- 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
- 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
- 2017Ensemble Distillation for Neural Machine Translationcited by 39
- 2020Language Model Prior for Low-Resource Neural Machine Translationcited by 37
- 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
- 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
- 2023MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methodscited by 33
- 2020A Study of Non-autoregressive Model for Sequence Generationcited by 31
- 2020Lifelong Language Knowledge Distillationcited by 31
- 2019Acquiring Knowledge from Pre-trained Model to Neural Machine Translationcited by 30
- 2020Future-Guided Incremental Transformer for Simultaneous Translationcited by 30
- 2021Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translationcited by 28
- 2022IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languagescited by 27
- 2016Neural Machine Translation with Pivot Languagescited by 25
- 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
- 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
- 2017A Teacher-Student Framework for Zero-Resource Neural Machine Translationcited by 22
50 most cited of 85 All 85 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.