mlsanityData to 7 Oct 2026

Ensemble Distillation for Neural Machine Translation

arXiv 1702.01802 ↗First posted 6 Feb 2017cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation describes a method for training a student network to perform better by learning from a stronger teacher network. Translating a sentence with an Neural Machine Translation (NMT) engine is time expensive and having a smaller model speeds up this process. We demonstrate how to transfer the translation quality of an ensemble and an oracle BLEU teacher network into a single NMT system. Further, we present translation improvements from a teacher network that has the same architecture and dimensions of the student network. As the training of the student model is still expensive, we introduce a data filtering method based on the knowledge of the teacher model that not only speeds up the training, but also leads to better translation quality. Our techniques need no code change and can be easily reproduced with any NMT architecture to speed up the decoding process.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 8; 2023: 9; 2024: 6; 2025: 1; 2026: 1.

39 papers.

Sort:

By area:

  1. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  2. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
  3. 2021Knowledge Distillation as Semiparametric Inferencecited by 29
  4. 2021Text Simplification by Taggingcited by 27
  5. 2021Maximum Bayes Smatch Ensemble Distillation for AMR Parsingcited by 27
  6. 2024Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLMcited by 26
  7. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  8. 2022One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translationcited by 20
  9. 2022On the benefits of knowledge distillation for adversarial robustnesscited by 17
  10. 2025TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judgecited by 16
  11. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
  12. 2021Multilingual AMR Parsing with Noisy Knowledge Distillationcited by 8
27 more
  1. 2021Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translationcited by 8
  2. 2023CUED at ProbSum 2023: Hierarchical Ensemble of Summarization Modelscited by 8
  3. 2023Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentationcited by 7
  4. 2022Unified and Effective Ensemble Knowledge Distillationcited by 5
  5. 2021The NiuTrans Machine Translation Systems for WMT21cited by 4
  6. 2021Amortized Noisy Channel Neural Machine Translationcited by 4
  7. 2022Twist Decoding: Diverse Generators Guide Each Othercited by 4
  8. 2022Continual Knowledge Distillation for Neural Machine Translationcited by 4
  9. 2023Accurate Knowledge Distillation with n-best Rerankingcited by 4
  10. 2023Minimum Bayes' Risk Decoding for System Combination of Grammatical Error Correction Systemscited by 4
  11. 2023Can a student Large Language Model perform as well as it's teacher?cited by 4
  12. 2021The Volctrans Neural Speech Translation System for IWSLT 2021cited by 3
  13. 2023Understanding Parameter Sharing in Transformerscited by 3
  14. 2021Sampling and Filtering of Neural Machine Translation Distillation Datacited by 2
  15. 2022Meta-Ensemble Parameter Learningcited by 2
  16. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  17. 2021Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translationcited by 1
  18. 2021Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairscited by 1
  19. 2023Heterogeneous-Branch Collaborative Learning for Dialogue Generationcited by 1
  20. 2023Leveraging Synthetic Targets for Machine Translationcited by 1
  21. 2024EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translationcited by 1
  22. 2024Erasure Coded Neural Network Inference via Fisher Averagingcited by 1
  23. 2021Hierarchical Knowledge Distillation for Dialogue Sequence Labelingcited by 0
  24. 2022Look Backward and Forward: Self-Knowledge Distillation with Bidirectional Decoder for Neural Machine Translationcited by 0
  25. 2023Pseudo-Label Training and Model Inertia in Neural Machine Translationcited by 0
  26. 2024Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealingcited by 0
  27. 2024Plug, Play, and Fuse: Zero-Shot Joint Decoding via Word-Level Re-ranking Across Diverse Vocabulariescited by 0