# Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

arXiv: https://arxiv.org/abs/2411.12240 · HTML: https://mlsanity.com/paper/2411.12240/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [Fast WordPiece Tokenization](https://mlsanity.com/paper/2012.15524/) (2020)
- [Getting the most out of your tokenizer for pre-training and domain adaptation](https://mlsanity.com/paper/2402.01035/) (2024)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages](https://mlsanity.com/paper/2203.05437/) (2022)
- [PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data](https://mlsanity.com/paper/2406.15053/) (2024)
- [A Formal Perspective on Byte-Pair Encoding](https://mlsanity.com/paper/2306.16837/) (2023)
- [INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages](https://mlsanity.com/paper/2407.13522/) (2024)
- [Theoretical Analysis of Byte-Pair Encoding](https://mlsanity.com/paper/2411.08671/) (2024)
- [Evaluating Various Tokenizers for Arabic Text Classification](https://mlsanity.com/paper/2106.07540/) (2021)
- [SUTRA: Scalable Multilingual Language Model Architecture](https://mlsanity.com/paper/2405.06694/) (2024)
- [Better Than Whitespace: Information Retrieval for Languages without Custom Tokenizers](https://mlsanity.com/paper/2210.05481/) (2022)
- [Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language](https://mlsanity.com/paper/2410.03718/) (2024)

## Built on by (most cited first, 11 of 11)

- [Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression](https://mlsanity.com/paper/2511.08066/) (2025)
- [Model-Aware Tokenizer Transfer](https://mlsanity.com/paper/2510.21954/) (2025)
- [MUTANT: A Recipe for Multilingual Tokenizer Design](https://mlsanity.com/paper/2511.03237/) (2025)
- [MacPrompt: Maraconic-guided Jailbreak against Text-to-Image Models](https://mlsanity.com/paper/2601.07141/) (2026)
- [The Model's Language Matters: A Comparative Privacy Analysis of LLMs](https://mlsanity.com/paper/2510.08813/) (2025)
- [PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relations](https://mlsanity.com/paper/2502.07186/) (2025)
- [Comparative Analysis of Tokenization Algorithms for Low-Resource Language Dzongkha](https://mlsanity.com/paper/2509.15255/) (2025)
- [How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers](https://mlsanity.com/paper/2601.11518/) (2026)
- [Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models](https://mlsanity.com/paper/2606.15044/) (2026)
- [SuTRA : Structurally-Unified Tokenization with Root Awareness](https://mlsanity.com/paper/2608.18087/) (2026)
- [A Grapheme-Aware Indic Tokenizer for Tamil: Large-Scale Training and Intrinsic Evaluation](https://mlsanity.com/paper/2609.06690/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) based on transformer architectures have revolutionized a variety of domains, with tokenization playing a pivotal role in their pre-processing and fine-tuning stages. In multilingual models, particularly those tailored for Indic languages, effective tokenization is crucial for optimizing performance. This paper presents a comprehensive evaluation of tokenizers used by 12 LLMs across all 22 official languages of India, with a focus on comparing the efficiency of their tokenization processes. We employed the Normalized Sequence Length (NSL) as a key metric in our analysis. Our findings reveal that the SUTRA tokenizer outperforms all other models, including several Indic-specific models, excelling in 14 languages. Notable insights include the SUTRA tokenizer's superior handling of Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing Indian languages, and the limited performance of Project Indus in certain languages. This study underscores the critical importance of developing targeted tokenization strategies for multilingual and Indic-centric models, laying the groundwork for future improvements in tokenizer design to enhance linguistic coverage and model efficiency.