# Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages

arXiv: https://arxiv.org/abs/2305.15814 · HTML: https://mlsanity.com/paper/2305.15814/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 27 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 9 of 9)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://mlsanity.com/paper/2010.14571/) (2020)
- [Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset](https://mlsanity.com/paper/2007.01176/) (2020)
- [Comparing Approaches to Dravidian Language Identification](https://mlsanity.com/paper/2103.05552/) (2021)

## Built on by (most cited first, 13 of 13)

- [IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages](https://mlsanity.com/paper/2403.06350/) (2024)
- [Fumbling in Babel: An Investigation into ChatGPT's Language Identification Ability](https://mlsanity.com/paper/2311.09696/) (2023)
- [Paramanu: Compact and Competitive Monolingual Language Models for Low-Resource Morphologically Rich Indian Languages](https://mlsanity.com/paper/2401.18034/) (2024)
- [Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real World Setting](https://mlsanity.com/paper/2512.10780/) (2025)
- [UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languages](https://mlsanity.com/paper/2509.21294/) (2025)
- [Pralekha: Cross-Lingual Document Alignment for Indic Languages](https://mlsanity.com/paper/2411.19096/) (2024)
- [Improving Informally Romanized Language Identification](https://mlsanity.com/paper/2504.21540/) (2025)
- [ILID: Native Script Language Identification for Indian Languages](https://mlsanity.com/paper/2507.11832/) (2025)
- [The Language of Bargaining: Linguistic Effects in LLM Negotiations](https://mlsanity.com/paper/2601.04387/) (2026)
- [How Should We Model the Probability of a Language?](https://mlsanity.com/paper/2602.08951/) (2026)
- [Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions](https://mlsanity.com/paper/2606.30790/) (2026)
- [Task-Conditional Flow Matching for Balanced Multilingual Text Embedding Adaptation](https://mlsanity.com/paper/2608.05785/) (2026)
- [BaatCheet: A Multilingual Corpus for Dialogue Translation in Indian Languages](https://mlsanity.com/paper/2609.33296/) (2026)

## Abstract (arXiv)

We create publicly available language identification (LID) datasets and models in all 22 Indian languages listed in the Indian constitution in both native-script and romanized text. First, we create Bhasha-Abhijnaanam, a language identification test set for native-script as well as romanized text which spans all 22 Indic languages. We also train IndicLID, a language identifier for all the above-mentioned languages in both native and romanized script. For native-script text, it has better language coverage than existing LIDs and is competitive or better than other LIDs. IndicLID is the first LID for romanized text in Indian languages. Two major challenges for romanized text LID are the lack of training data and low-LID performance when languages are similar. We provide simple and effective solutions to these problems. In general, there has been limited work on romanized text in any language, and our findings are relevant to other languages that need romanized language identification. Our models are publicly available at https://ai4bharat.iitm.ac.in/indiclid under open-source licenses. Our training and test sets are also publicly available at https://ai4bharat.iitm.ac.in/bhasha-abhijnaanam under open-source licenses.