Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages
Not on the map. Why?
Abstract (arXiv)
With multilingual machine translation (MMT) models continuing to grow in size and number of supported languages, it is natural to reuse and upgrade existing models to save computation as data becomes available in more languages. However, adding new languages requires updating the vocabulary, which complicates the reuse of embeddings. The question of how to reuse existing models while also making architectural changes to provide capacity for both old and new languages has also not been closely studied. In this work, we introduce three techniques that help speed up effective learning of the new languages and alleviate catastrophic forgetting despite vocabulary and architecture mismatches. Our results show that by (1) carefully initializing the network, (2) applying learning rate scaling, and (3) performing data up-sampling, it is possible to exceed the performance of a same-sized baseline model with 30% computation and recover the performance of a larger model trained from scratch with over 50% reduction in computation. Furthermore, our analysis reveals that the introduced techniques help learn the new directions more effectively and alleviate catastrophic forgetting at the same time. We hope our work will guide research into more efficient approaches to growing languages for these MMT models and ultimately maximize the reuse of existing models.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 7 Feb 2023.
Builds on
27 of 40 references.
References · 27
- 2017Attention Is All You Needcited by 59,932
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2016Progressive Neural Networkscited by 1,332
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
- 2020Beyond English-Centric Multilingual Machine Translationcited by 559
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
- 2019Massively Multilingual Neural Machine Translationcited by 246
- 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
- 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
- 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
15 more
- 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
- 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
- 2016Toward Multilingual Neural Machine Translation with Universal Encoder and Decodercited by 76
- 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
- 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 61
- 2022Architecture Matters in Continual Learningcited by 46
- 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
- 2020Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoderscited by 23
- 2018Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabularycited by 18
- 2020Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translationcited by 17
- 2022SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languagescited by 16
- 2019From Bilingual to Multilingual Neural Machine Translation by Incremental Trainingcited by 14
- 2019Adapting Multilingual Neural Machine Translation to Unseen Languagescited by 12
- 2021Continual Learning in Multilingual NMT via Language-Specific Embeddingscited by 12
- 2020Exploring Benefits of Transfer Learning in Neural Machine Translationcited by 5
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 1.
2 papers.