# Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages

arXiv: https://arxiv.org/abs/2608.07727 · HTML: https://mlsanity.com/paper/2608.07727/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://mlsanity.com/paper/2104.05596/) (2021)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)

## Abstract (arXiv)

Dravidian languages, mainly Tamil, Telugu, Kannada, and Malayalam make up only a small part of the data used to train multilingual language models, so it's not clear how much per-language ability these models actually keep. I have trained five GPT-2 architecture models from scratch to compare four monolingual models (one each for Tamil, Telugu, Kannada, and Malayalam, each with its own 32K-vocabulary subword tokenizer) against one multilingual model sharing a 64K-vocabulary subword tokenizer across all four languages. All the 5 models are trained on cleaned CC-100, Wikipedia, and Samanantar data. I have tested the models on perplexity, bits-per-byte, tokenizer efficiency, and fine-tuning results which are compared against mGPT. The monolingual models outperform mGPT on sentiment classification and named entity recognition, and their tokenizers proved more efficient than the shared multilingual model across all the languages tested.