BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
Not on the map. Why?
Abstract (arXiv)
Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages. Sanskrit, Tamil, and other classical Indic languages exhibit agglutinative morphology, productive sandhi (phonological fusion at word boundaries), and domain-specific vocabularies absent from general-purpose training data. This paper presents BHARATI, a set of SentencePiece BPE tokenizers trained on a balanced 781 MB corpus spanning seven languages (English, Hindi, Sanskrit, Tamil, Telugu, Kannada, and Malayalam) with native script support for all languages. We describe three successive tokenizer versions: v1 (English and Sanskrit only, with broken byte-fallback for Tamil), v2 (four-language support with byte-level fallback for southern languages), and v3 (full seven-language native subword coverage). Subword fertility analysis demonstrates that v3 averages 2.6 tokens per Indian Knowledge System (IKS) technical term, compared to 5.25 tokens per term with GPT-2's tokenizer and 3.75 tokens with the multilingual SentencePiece baseline, with the largest gains on a set of reserved IKS terms that are represented as single tokens by construction. On a held-out test set of 490 IKS-domain sentences (70 per language across seven languages, released with the measurement script), v3 reduces sequence length by roughly 90% relative to GPT-2 and byte-level encoding (which lack native Indic subwords) and by approximately 25% relative to the mBART-50 multilingual baseline, averaged across the six Indic languages, directly translating to increased effective context length for downstream language models. The tokenizer models (32,000 vocabulary), training scripts, and evaluation benchmarks are released under open licenses.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 42 references that are in the map. First posted on arXiv 25 Jul 2026.
Builds on
18 of 42 references.
References · 18
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
- 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
- 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
- 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 61
- 2020AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languagescited by 43
- 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 30
- 2024Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removalcited by 16
6 more
- 2024One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Taskscited by 15
- 2025Tokenization Matters: Improving Zero-Shot NER for Indic Languagescited by 12
- 2025Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Taskscited by 9
- 2023SanskritShala: A Neural Sanskrit NLP Toolkit with Web-Based Interface for Pedagogical and Annotation Purposescited by 5
- 2023Tokenization with Factorized Subword Encodingcited by 4
- 2025MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Languagecited by 1
Built on by
Not cited yet.