# MuRIL: Multilingual Representations for Indian Languages arXiv: https://arxiv.org/abs/2103.10730 ยท HTML: https://mlsanity.com/paper/2103.10730/ ## Key facts - Cited by 179 papers in the map (data through 14 Oct 2026). - Builds on 10 of its 19 references that are in the map. - First posted on arXiv 19 Mar 2021. ## Builds on (most cited first, 10 of 10) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019) - [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020) - [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019) - [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020) - [Improving Multilingual Models with Language-Clustered Vocabularies](https://mlsanity.com/paper/2010.12777/) (2020) - [PMIndia -- A Collection of Parallel Corpora of Languages of India](https://mlsanity.com/paper/2001.09907/) (2020) - [Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset](https://mlsanity.com/paper/2007.01176/) (2020) - [FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding](https://mlsanity.com/paper/2009.05166/) (2020) ## Built on by (most cited first, 30 of 179) - [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024) - [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023) - [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023) - [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023) - [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021) - [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022) - [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022) - [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021) - [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024) - [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022) - [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021) - [IndicBART: A Pre-trained Model for Indic Natural Language Generation](https://mlsanity.com/paper/2109.02903/) (2021) - [L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages](https://mlsanity.com/paper/2211.11418/) (2022) - [IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context](https://mlsanity.com/paper/2403.20147/) (2024) - [Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages](https://mlsanity.com/paper/2212.10168/) (2022) - [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022) - [IndicXNLI: Evaluating Multilingual Inference for Indian Languages](https://mlsanity.com/paper/2204.08776/) (2022) - [Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages](https://mlsanity.com/paper/2203.01976/) (2022) - [Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages](https://mlsanity.com/paper/2109.10534/) (2021) - [Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus](https://mlsanity.com/paper/2410.14815/) (2024) - [Data Bootstrapping Approaches to Improve Low Resource Abusive Language Detection for Indic Languages](https://mlsanity.com/paper/2204.12543/) (2022) - [Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages](https://mlsanity.com/paper/2205.12676/) (2022) - [Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study](https://mlsanity.com/paper/2106.03958/) (2021) - [Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users](https://mlsanity.com/paper/2205.03018/) (2022) - [Hate Speech and Offensive Language Detection in Bengali](https://mlsanity.com/paper/2210.03479/) (2022) - [ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment](https://mlsanity.com/paper/2305.14463/) (2023) - [BanglaAbuseMeme: A Dataset for Bengali Abusive Meme Classification](https://mlsanity.com/paper/2310.11748/) (2023) - [SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2)](https://mlsanity.com/paper/2305.06586/) (2023) - [The Case for "Thick Evaluations" of Cultural Representation in AI](https://mlsanity.com/paper/2503.19075/) (2025) - [L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT](https://mlsanity.com/paper/2304.11434/) (2023) ## Abstract (arXiv) India is a multilingual society with 1369 rationalized languages and dialects being spoken across the country (INDIA, 2011). Of these, the 22 scheduled languages have a staggering total of 1.17 billion speakers and 121 languages have more than 10,000 speakers (INDIA, 2011). India also has the second largest (and an ever growing) digital footprint (Statista, 2020). Despite this, today's state-of-the-art multilingual systems perform suboptimally on Indian (IN) languages. This can be explained by the fact that multilingual language models (LMs) are often trained on 100+ languages together, leading to a small representation of IN languages in their vocabulary and training data. Multilingual LMs are substantially less effective in resource-lean scenarios (Wu and Dredze, 2020; Lauscher et al., 2020), as limited data doesn't help capture the various nuances of a language. One also commonly observes IN language text transliterated to Latin or code-mixed with English, especially in informal settings (for example, on social media platforms) (Rijhwani et al., 2017). This phenomenon is not adequately handled by current state-of-the-art multilingual LMs. To address the aforementioned gaps, we propose MuRIL, a multilingual LM specifically built for IN languages. MuRIL is trained on significantly large amounts of IN text corpora only. We explicitly augment monolingual text corpora with both translated and transliterated document pairs, that serve as supervised cross-lingual signals in training. MuRIL significantly outperforms multilingual BERT (mBERT) on all tasks in the challenging cross-lingual XTREME benchmark (Hu et al., 2020). We also present results on transliterated (native to Latin script) test sets of the chosen datasets and demonstrate the efficacy of MuRIL in handling transliterated data.