mlsanityData to 7 Oct 2026

Does Transliteration Help Multilingual Language Modeling?

arXiv 2201.12501 ↗First posted 29 Jan 2022cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Script diversity presents a challenge to Multilingual Language Models (MLLM) by reducing lexical overlap among closely related languages. Therefore, transliterating closely related languages that use different writing scripts to a common script may improve the downstream task performance of MLLMs. We empirically measure the effect of transliteration on MLLMs in this context. We specifically focus on the Indic languages, which have the highest script diversity in the world, and we evaluate our models on the IndicGLUE benchmark. We perform the Mann-Whitney U test to rigorously verify whether the effect of transliteration is significant or not. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages. We also measure the cross-lingual representation similarity of the models using centered kernel alignment on parallel sentences from the FLORES-101 dataset. We find that for parallel sentences across different languages, the transliteration-based model learns sentence representations that are more similar.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 28 of its 72 references that are in the map. First posted on arXiv 29 Jan 2022.

Builds on

28 of 72 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  8. 2019How multilingual is Multilingual BERT?cited by 642
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  11. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  12. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
16 more
  1. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  2. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  3. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  4. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  5. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  6. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  7. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  8. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  9. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  10. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  11. 2019Pushing the Limits of Low-Resource Morphological Inflectioncited by 29
  12. 2021Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languagescited by 24
  13. 2021Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Studycited by 19
  14. 2018Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representationscited by 18
  15. 2018Zero-shot Neural Transfer for Cross-lingual Entity Linkingcited by 11
  16. 2020On Romanization for Model Transfer Between Scripts in Neural Machine Translationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 6; 2025: 1; 2026: 4.

14 papers.

Sort:

  1. 2022Language Modelling with Pixelscited by 53
  2. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  3. 2023Romanization-based Large-scale Adaptation of Multilingual Language Modelscited by 21
  4. 2024Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignmentcited by 14
  5. 2024Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansioncited by 4
  6. 2023Hate Speech and Offensive Content Detection in Indo-Aryan Languages: A Battle of LSTM and Transformerscited by 3
  7. 2024Milestones in Bengali Sentiment Analysis leveraging Transformer-models: Fundamentals, Challenges and Future Directionscited by 1
  8. 2024Training a Bilingual Language Model by Mapping Tokens onto a Shared Character Spacecited by 1
  9. 2025Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languagescited by 1
  10. 2026One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Modelscited by 1
  11. 2026One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representationscited by 1
  12. 2024Script-Agnostic Language Identificationcited by 0
2 more
  1. 2026A Study of Crosslinguistic Influence in Language Modelscited by 0
  2. 2026Why Pretraining Fails to Share Cross-Lingual Knowledgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024How Transliterations Improve Crosslingual Alignment
  2. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
  3. 2024Beyond Data Quantity: Key Factors Driving Performance in Multilingual Language Models
  4. 2024Multilingual Large Language Models: A Systematic Survey
  5. 2024What Drives Performance in Multilingual Language Models?
  6. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers
  7. 2024Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact
  8. 2024Pruning Multilingual Large Language Models for Multilingual Inference