Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights
Not on the map. Why?
Abstract (arXiv)
Tokenization plays a pivotal role in NLP and is fundamental to training language models. However, existing tokenizers are often skewed towards high-resource languages, limiting their effectiveness for linguistically diverse and morphologically rich languages such as those in the Indian subcontinent. In this work, we present a comprehensive empirical study of multilingual tokenization across 17 Indic languages spanning 11 scripts and two language families. We systematically evaluate the effects of (i) widely used subword algorithms: BPE and Unigram LM, (ii) script and orthography-aware normalization, (iii) vocabulary size, and (iv) multilingual vocabulary construction strategies. We use a combination of intrinsic and extrinsic evaluations to obtain the following observations: (i) script-specific normalization improves tokenization quality, (ii) Unigram LM better preserves morphological boundaries than BPE, (iii) cluster-based vocabulary construction shows improvement in downstream tasks compared to the joint method. Our findings highlight the importance of linguistically informed design choices in multilingual tokenization and offer practical guidance for building effective tokenizers for low-resource and morphologically complex languages.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 21 Jun 2025.
Builds on
21 of 32 references.
References · 21
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
- 2019Cross-lingual Language Model Pretrainingcited by 1,090
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
- 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
- 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
- 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
- 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
- 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
- 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 62
9 more
- 2024Why do language models perform worse for morphologically complex languages?cited by 62
- 2023Tokenization and the Noiseless Channelcited by 59
- 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
- 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
- 2024Airavata: Introducing Hindi Instruction-tuned LLMcited by 32
- 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 31
- 2024Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removalcited by 16
- 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
- 2025Beyond Literal Token Overlap: Token Alignability for Multilingualitycited by 9
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.