# Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset

arXiv: https://arxiv.org/abs/2007.01176 · HTML: https://mlsanity.com/paper/2007.01176/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2020.

## Built on by (most cited first, 30 of 35)

- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots](https://mlsanity.com/paper/2103.09593/) (2021)
- [BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLP](https://mlsanity.com/paper/2309.13173/) (2023)
- [Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages](https://mlsanity.com/paper/2205.12676/) (2022)
- [Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users](https://mlsanity.com/paper/2205.03018/) (2022)
- [Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages](https://mlsanity.com/paper/2305.15814/) (2023)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency](https://mlsanity.com/paper/2306.04530/) (2023)
- [BnSentMix: A Diverse Bengali-English Code-Mixed Dataset for Sentiment Analysis](https://mlsanity.com/paper/2408.08964/) (2024)
- [Hate-Alert@DravidianLangTech-EACL2021: Ensembling strategies for Transformer-based Offensive language Detection](https://mlsanity.com/paper/2102.10084/) (2021)
- [Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs](https://mlsanity.com/paper/2407.00416/) (2024)
- [Do Images really do the Talking? Analysing the significance of Images in Tamil Troll meme classification](https://mlsanity.com/paper/2108.03886/) (2021)
- [User-Aware Multilingual Abusive Content Detection in Social Media](https://mlsanity.com/paper/2410.21321/) (2024)
- [Sinhala Transliteration: A Comparative Analysis Between Rule-based and Seq2Seq Approaches](https://mlsanity.com/paper/2501.00529/) (2024)
- [Generalists vs. Specialists: Evaluating Large Language Models for Urdu](https://mlsanity.com/paper/2407.04459/) (2024)
- [Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Posts](https://mlsanity.com/paper/2108.10939/) (2021)
- [IndoNLP 2025: Shared Task on Real-Time Reverse Transliteration for Romanized Indo-Aryan languages](https://mlsanity.com/paper/2501.05816/) (2025)
- [A Survey of Pun Generation: Datasets, Evaluations and Methodologies](https://mlsanity.com/paper/2507.04793/) (2025)
- [Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction](https://mlsanity.com/paper/2106.12698/) (2021)
- [Gui at MixMT 2022 : English-Hinglish: An MT approach for translation of code mixed data](https://mlsanity.com/paper/2210.12215/) (2022)
- [Spelling convention sensitivity in neural language models](https://mlsanity.com/paper/2303.03457/) (2023)
- [IruMozhi: Automatically classifying diglossia in Tamil](https://mlsanity.com/paper/2311.07804/) (2023)
- [Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models](https://mlsanity.com/paper/2503.21530/) (2025)
- [Improving Informally Romanized Language Identification](https://mlsanity.com/paper/2504.21540/) (2025)
- [Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integration](https://mlsanity.com/paper/2511.22769/) (2025)
- [Detection of Emotions in Hindi-English Code Mixed Text Data](https://mlsanity.com/paper/2105.09226/) (2021)

## Abstract (arXiv)

This paper describes the Dakshina dataset, a new resource consisting of text in both the Latin and native scripts for 12 South Asian languages. The dataset includes, for each language: 1) native script Wikipedia text; 2) a romanization lexicon; and 3) full sentence parallel data in both a native script of the language and the basic Latin alphabet. We document the methods used for preparation and selection of the Wikipedia text in each language; collection of attested romanizations for sampled lexicons; and manual romanization of held-out sentences from the native script collections. We additionally provide baseline results on several tasks made possible by the dataset, including single word transliteration, full sentence transliteration, and language modeling of native script and romanized text. Keywords: romanization, transliteration, South Asian languages