mlsanityData to 7 Oct 2026

RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization

arXiv 2401.14280 ↗First posted 25 Jan 2024cs.CL cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages that use non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Our approach involves the continual pretraining of an English LLM like Llama 2 on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches or outperforms native script representation across various NLU, NLG, and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP. Our code is available on https://github.com/AI4Bharat/romansetu.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

36 of 56 references.

References · 36

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
24 more
  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  4. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  5. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  6. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  7. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  8. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  9. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  10. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  11. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  12. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
  13. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  14. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  15. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  16. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  17. 2022IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languagescited by 42
  18. 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
  19. 2021Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languagescited by 24
  20. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  21. 2023Romanization-based Large-scale Adaptation of Multilingual Language Modelscited by 21
  22. 2021Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Studycited by 19
  23. 2022Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Userscited by 19
  24. 2022Does Transliteration Help Multilingual Language Modeling?cited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 10; 2026: 11.

29 papers.

Sort:

By area:

  1. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  2. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  3. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  4. 2025RomanLens: The Role Of Latent Romanization In Multilinguality In LLMscited by 11
  5. 2024TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Datacited by 9
  6. 2025TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarkingcited by 9
  7. 2024Facilitating large language model Russian adaptation with Learned Embedding Propagationcited by 8
  8. 2024LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Languagecited by 6
  9. 2024Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languagescited by 6
  10. 2025Overcoming Vocabulary Constraints with Pixel-level Fallbackcited by 6
  11. 2025Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challengescited by 6
  12. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
17 more
  1. 2025Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real World Settingcited by 5
  2. 2024Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?cited by 2
  3. 2025Happiness is Sharing a Vocabulary: A Study of Transliteration Methodscited by 2
  4. 2025Uncovering inequalities in new knowledge learning by large language models across different languagescited by 1
  5. 2025PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generationcited by 1
  6. 2026One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Modelscited by 1
  7. 2025Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of English-Japanese Biomedical Adaptationcited by 0
  8. 2026Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliterationcited by 0
  9. 2026Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhalacited by 0
  10. 2026BhashaSetu: A Data-Centric Approach to Low-Resource Machine Translationcited by 0
  11. 2026The Latin Substrate: How Language Models Represent and Mediate Script Choicecited by 0
  12. 2026Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructionscited by 0
  13. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognitioncited by 0
  14. 2026One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthographycited by 0
  15. 2026VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languagescited by 0
  16. 2026Why Pretraining Fails to Share Cross-Lingual Knowledgecited by 0
  17. 2026Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Termscited by 0