mlsanityData to 14 Oct 2026

Multilingual Neural Machine Translation:Can Linguistic Hierarchies Help?

arXiv 2110.07816 ↗First posted 15 Oct 2021cs.CL cs.AIcited by 2

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Multilingual Neural Machine Translation (MNMT) trains a single NMT model that supports translation between multiple languages, rather than training separate models for different languages. Learning a single model can enhance the low-resource translation by leveraging data from multiple languages. However, the performance of an MNMT model is highly dependent on the type of languages used in training, as transferring knowledge from a diverse set of languages degrades the translation performance due to negative transfer. In this paper, we propose a Hierarchical Knowledge Distillation (HKD) approach for MNMT which capitalises on language groups generated according to typological features and phylogeny of languages to overcome the issue of negative transfer. HKD generates a set of multilingual teacher-assistant models via a selective knowledge distillation mechanism based on the language groups, and then distils the ultimate multilingual model from those assistants in an adaptive way. Experimental results derived from the TED dataset with 53 languages demonstrate the effectiveness of our approach in avoiding the negative transfer effect in MNMT, leading to an improved translation performance (about 1 BLEU score on average) compared to strong baselines.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 21 of its 35 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

21 of 35 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2015Generating Sentences from a Continuous Spacecited by 612
  7. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  8. 2017Six Challenges for Neural Machine Translationcited by 335
  9. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  10. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 126
  11. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  12. 2016Toward Multilingual Neural Machine Translation with Universal Encoder and Decodercited by 76
9 more
  1. 2020Balancing Training for Multilingual Neural Machine Translationcited by 72
  2. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  3. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  4. 2019Multilingual Neural Machine Translation with Language Clusteringcited by 47
  5. 2018A neural interlingua for multilingual machine translationcited by 29
  6. 2020Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representationscited by 21
  7. 2019What do Language Representations Really Represent?cited by 17
  8. 2016Survey on the Use of Typological Information in Natural Language Processingcited by 6
  9. 2020Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  2. 2024Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiencescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models
  2. 2021Selective Knowledge Distillation for Neural Machine Translation
  3. 2022Unifying the Convergences in Multilingual Neural Machine Translation
  4. 2026Evolving Knowledge Distillation for Lightweight Neural Machine Translation
  5. 2025Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages
  6. 2022Life-long Learning for Multilingual Neural Machine Translation with Knowledge Distillation
  7. 2024Enhancing Low-Resource NMT with a Multilingual Encoder and Knowledge Distillation: A Case Study
  8. 2023Unraveling Key Factors of Knowledge Distillation