mlsanityData to 14 Oct 2026

Continual Learning in Multilingual NMT via Language-Specific Embeddings

arXiv 2110.10478 ↗First posted 20 Oct 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper proposes a technique for adding a new source or target language to an existing multilingual NMT model without re-training it on the initial set of languages. It consists in replacing the shared vocabulary with a small language-specific vocabulary and fine-tuning the new embeddings on the new language's parallel data. Some additional language-specific components may be trained to improve performance (e.g., Transformer layers or adapter modules). Because the parameters of the original model are not modified, its performance on the initial languages does not degrade. We show on two sets of experiments (small-scale on TED Talks, and large-scale on ParaCrawl) that this approach performs as well or better as the more costly alternatives; and that it has excellent zero-shot performance: training on English-centric data is enough to translate between the new language and any of the initial languages.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 20 Oct 2021.

Builds on

22 of 30 references.

References · 22

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  4. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  5. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  6. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
  7. 2019Massively Multilingual Neural Machine Translationcited by 246
  8. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  9. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 126
  10. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  11. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  12. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
10 more
  1. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  2. 2020Complete Multilingual Neural Machine Translationcited by 30
  3. 2019Naver Labs Europe's Systems for the WMT19 Machine Translation Robustness Taskcited by 23
  4. 2020Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoderscited by 23
  5. 2019From Bilingual to Multilingual Neural Machine Translation by Incremental Trainingcited by 14
  6. 2019Adapting Multilingual Neural Machine Translation to Unseen Languagescited by 12
  7. 2018Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translationcited by 10
  8. 2021Efficient Inference for Multilingual Neural Machine Translationcited by 10
  9. 2020Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoderscited by 8
  10. 2020Revisiting Modularized Multilingual NMT to Meet Industrial Demandscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 3; 2024: 3; 2025: 1.

12 papers.

  1. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  2. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  3. 2025Conditions for Catastrophic Forgetting in Multilingual Translationcited by 10
  4. 2022Parameter-Efficient Finetuning for Robust Continual Multilingual Learningcited by 9
  5. 2023NAVER LABS Europe's Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Trackcited by 9
  6. 2022Adapting BigScience Multilingual Model to Unseen Languagescited by 6
  7. 2022Continual Learning of Neural Machine Translation within Low Forgetting Risk Regionscited by 6
  8. 2024Exploring Continual Learning of Compositional Generalization in NLIcited by 4
  9. 2024Self-generated Replay Memories for Continual Neural Machine Translationcited by 4
  10. 2023Extending Multilingual Machine Translation through Imitation Learningcited by 3
  11. 2023Efficiently Upgrading Multilingual Machine Translation Models to Support More Languagescited by 2
  12. 2022Impact of Domain-Adapted Multilingual Neural Machine Translation in the Medical Domaincited by 0