# Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integration

arXiv: https://arxiv.org/abs/2511.22769 · HTML: https://mlsanity.com/paper/2511.22769/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Cross-Entropy Loss Functions: Theoretical Analysis and Applications](https://mlsanity.com/paper/2304.07288/) (2023)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [The IIT Bombay English-Hindi Parallel Corpus](https://mlsanity.com/paper/1710.02855/) (2017)
- [Does BLEU Score Work for Code Migration?](https://mlsanity.com/paper/1906.04903/) (2019)
- [BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla](https://mlsanity.com/paper/2205.11081/) (2022)
- [Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset](https://mlsanity.com/paper/2007.01176/) (2020)
- [BanTH: A Multi-label Hate Speech Detection Dataset for Transliterated Bangla](https://mlsanity.com/paper/2410.13281/) (2024)
- [IndoNLP 2025: Shared Task on Real-Time Reverse Transliteration for Romanized Indo-Aryan languages](https://mlsanity.com/paper/2501.05816/) (2025)
- [Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation](https://mlsanity.com/paper/2509.00973/) (2025)
- [End-to-End Natural Language Understanding Pipeline for Bangla Conversational Agents](https://mlsanity.com/paper/2107.05541/) (2021)
- [UAST: Unicode Aware Sanskrit Transliteration](https://mlsanity.com/paper/2203.14277/) (2022)
- [Lost in Transliteration: Bridging the Script Gap in Neural IR](https://mlsanity.com/paper/2505.08411/) (2025)

## Built on by (most cited first, 1 of 1)

- [Multilingual Multi-Label Emotion Classification at Scale with Synthetic Data](https://mlsanity.com/paper/2604.12633/) (2026)

## Abstract (arXiv)

The development of robust transliteration techniques to enhance the effectiveness of transforming Romanized scripts into native scripts is crucial for Natural Language Processing tasks, including sentiment analysis, speech recognition, information retrieval, and intelligent personal assistants. Despite significant advancements, state-of-the-art multilingual models still face challenges in handling Romanized script, where the Roman alphabet is adopted to represent the phonetic structure of diverse languages. Within the South Asian context, where the use of Romanized script for Indo-Aryan languages is widespread across social media and digital communication platforms, such usage continues to pose significant challenges for cutting-edge multilingual models. While a limited number of transliteration datasets and models are available for Indo-Aryan languages, they generally lack sufficient diversity in pronunciation and spelling variations, adequate code-mixed data for large language model (LLM) training, and low-resource adaptation. To address this research gap, we introduce a novel transliteration dataset for two popular Indo-Aryan languages, Hindi and Bengali, which are ranked as the 3rd and 7th most spoken languages worldwide. Our dataset comprises nearly 1.8 million Hindi and 1 million Bengali transliteration pairs. In addition to that, we pre-train a custom multilingual seq2seq LLM based on Marian architecture using the developed dataset. Experimental results demonstrate significant improvements compared to existing relevant models in terms of BLEU and CER metrics.