mlsanityData to 14 Oct 2026

Romanization-based Large-scale Adaptation of Multilingual Language Models

arXiv 2304.08865 ↗First posted 18 Apr 2023cs.CL cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Large multilingual pretrained language models (mPLMs) have become the de facto state of the art for cross-lingual transfer in NLP. However, their large-scale deployment to many languages, besides pretraining data scarcity, is also hindered by the increase in vocabulary size and limitations in their parameter budget. In order to boost the capacity of mPLMs to deal with low-resource and unseen languages, we explore the potential of leveraging transliteration on a massive scale. In particular, we explore the UROMAN transliteration tool, which provides mappings from UTF-8 to Latin characters for all the writing systems, enabling inexpensive romanization for virtually any language. We first focus on establishing how UROMAN compares against other language-specific and manually curated transliterators for adapting multilingual PLMs. We then study and compare a plethora of data- and parameter-efficient strategies for adapting the mPLMs to romanized and non-romanized corpora of 14 diverse low-resource languages. Our results reveal that UROMAN-based transliteration can offer strong performance for many languages, with particular gains achieved in the most challenging setups: on languages with unseen scripts and with limited training data without any vocabulary augmentation. Further analyses reveal that an improved tokenizer based on romanized data can even outperform non-transliteration-based methods in the majority of languages.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 18 Apr 2023.

Builds on

25 of 42 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  7. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  8. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  9. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  10. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  12. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
13 more
  1. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  2. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  3. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  4. 2023Modular Deep Learningcited by 83
  5. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  6. 2020Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigationcited by 34
  7. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  8. 2021Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languagescited by 24
  9. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  10. 2022MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialoguecited by 15
  11. 2022Does Transliteration Help Multilingual Language Modeling?cited by 14
  12. 2020Introducing various Semantic Models for Amharic: Experimentation and Evaluation with multiple Tasks and Datasetscited by 12
  13. 2020On Romanization for Model Transfer Between Scripts in Neural Machine Translationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 3; 2026: 7.

21 papers.

Sort:

  1. 2024Teaching Large Language Models an Unseen Language on the Flycited by 30
  2. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  3. 2024Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignmentcited by 14
  4. 2024TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Datacited by 9
  5. 2024Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scriptscited by 8
  6. 2023Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalizationcited by 6
  7. 2024How Transliterations Improve Crosslingual Alignmentcited by 6
  8. 2024TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Modelscited by 3
  9. 2024Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptationcited by 3
  10. 2025Happiness is Sharing a Vocabulary: A Study of Transliteration Methodscited by 2
  11. 2024Training a Bilingual Language Model by Mapping Tokens onto a Shared Character Spacecited by 1
  12. 2024Romanization Encoding For Multilingual ASRcited by 1
9 more
  1. 2025Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languagescited by 1
  2. 2026One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Modelscited by 1
  3. 2026Pingala: Prosody-Aware Decoding for Sanskrit Poetry Generationcited by 1
  4. 2025Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of English-Japanese Biomedical Adaptationcited by 0
  5. 2026Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliterationcited by 0
  6. 2026A Study of Crosslinguistic Influence in Language Modelscited by 0
  7. 2026The Latin Substrate: How Language Models Represent and Mediate Script Choicecited by 0
  8. 2026Cross-Lingual Transfer for Machine Translation in Turkic Languagescited by 0
  9. 2026One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthographycited by 0