Extending the Subwording Model of Multilingual Pretrained Models for New Languages
Not on the map. Why?
Abstract (arXiv)
Multilingual pretrained models are effective for machine translation and cross-lingual processing because they contain multiple languages in one model. However, they are pretrained after their tokenizers are fixed; therefore it is difficult to change the vocabulary after pretraining. When we extend the pretrained models to new languages, we must modify the tokenizers simultaneously. In this paper, we add new subwords to the SentencePiece tokenizer to apply a multilingual pretrained model to new languages (Inuktitut in this paper). In our experiments, we segmented Inuktitut sentences into subwords without changing the segmentation of already pretrained languages, and applied the mBART-50 pretrained model to English-Inuktitut translation.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 29 Nov 2022.
Builds on
21 of 33 references.
References · 21
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
9 more
- 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
- 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
- 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
- 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
- 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
- 2020Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translationcited by 38
- 2020Facebook AI's WMT20 News Translation Task Submissioncited by 17
- 2021Joint Optimization of Tokenization and Downstream Modelcited by 15
- 2021The NiuTrans Machine Translation Systems for WMT21cited by 4
Built on by
Citing papers per year: 2023: 1; 2024: 2.
3 papers.