mlsanityData to 7 Oct 2026

Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset

arXiv 2007.01176 ↗First posted 2 Jul 2020cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

This paper describes the Dakshina dataset, a new resource consisting of text in both the Latin and native scripts for 12 South Asian languages. The dataset includes, for each language: 1) native script Wikipedia text; 2) a romanization lexicon; and 3) full sentence parallel data in both a native script of the language and the basic Latin alphabet. We document the methods used for preparation and selection of the Wikipedia text in each language; collection of attested romanizations for sampled lexicons; and manual romanization of held-out sentences from the native script collections. We additionally provide baseline results on several tasks made possible by the dataset, including single word transliteration, full sentence transliteration, and language modeling of native script and romanized text. Keywords: romanization, transliteration, South Asian languages

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 4; 2023: 7; 2024: 5; 2025: 6; 2026: 4.

35 papers.

Sort:

  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  2. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  4. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  5. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  6. 2021Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglotscited by 23
  7. 2023BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLPcited by 23
  8. 2022Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languagescited by 20
  9. 2022Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Userscited by 19
  10. 2023Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languagescited by 13
  11. 2023Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistencycited by 10
  12. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 10
23 more
  1. 2024BnSentMix: A Diverse Bengali-English Code-Mixed Dataset for Sentiment Analysiscited by 9
  2. 2021Hate-Alert@DravidianLangTech-EACL2021: Ensembling strategies for Transformer-based Offensive language Detectioncited by 8
  3. 2021Do Images really do the Talking? Analysing the significance of Images in Tamil Troll meme classificationcited by 6
  4. 2024Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMscited by 6
  5. 2024User-Aware Multilingual Abusive Content Detection in Social Mediacited by 6
  6. 2024Sinhala Transliteration: A Comparative Analysis Between Rule-based and Seq2Seq Approachescited by 6
  7. 2024Generalists vs. Specialists: Evaluating Large Language Models for Urducited by 5
  8. 2021Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Postscited by 4
  9. 2025IndoNLP 2025: Shared Task on Real-Time Reverse Transliteration for Romanized Indo-Aryan languagescited by 3
  10. 2025A Survey of Pun Generation: Datasets, Evaluations and Methodologiescited by 2
  11. 2021Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transductioncited by 1
  12. 2022Gui at MixMT 2022 : English-Hinglish: An MT approach for translation of code mixed datacited by 1
  13. 2023Spelling convention sensitivity in neural language modelscited by 1
  14. 2023IruMozhi: Automatically classifying diglossia in Tamilcited by 1
  15. 2025Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Modelscited by 1
  16. 2025Improving Informally Romanized Language Identificationcited by 1
  17. 2025Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integrationcited by 1
  18. 2021Detection of Emotions in Hindi-English Code Mixed Text Datacited by 0
  19. 2023Structure-Aware Path Inference for Neural Finite State Transducerscited by 0
  20. 2025Beyond Specialization: Benchmarking LLMs for Transliteration of Indian Languagescited by 0
  21. 2026Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliterationcited by 0
  22. 2026Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhalacited by 0
  23. 2026VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languagescited by 0