Unknown Script: Impact of Script on Cross-Lingual Transfer
Not on the map. Why?
Abstract (arXiv)
Cross-lingual transfer has become an effective way of transferring knowledge between languages. In this paper, we explore an often overlooked aspect in this domain: the influence of the source language of a language model on language transfer performance. We consider a case where the target language and its script are not part of the pre-trained model. We conduct a series of experiments on monolingual and multilingual models that are pre-trained on different tokenization methods to determine factors that affect cross-lingual transfer to a new language with a unique script. Our findings reveal the importance of the tokenizer as a stronger factor than the shared script, language similarity, and model size.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 29 Apr 2024.
Builds on
20 of 29 references.
References · 20
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
- 2019How multilingual is Multilingual BERT?cited by 642
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
- 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
- 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
- 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
- 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
8 more
- 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
- 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
- 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
- 2020Multilingual Alignment of Contextual Word Representationscited by 113
- 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
- 2023Spanish Pre-trained BERT Model and Evaluation Datacited by 81
- 2021Classifying Argumentative Relations Using Logical Mechanisms and Argumentation Schemescited by 17
- 2021Contemporary Amharic Corpus: Automatically Morpho-Syntactically Tagged Amharic Corpuscited by 4
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.