BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
Not on the map. Why?
Abstract (arXiv)
We present BrahmicTokenizer-131K, a 131,072-vocabulary byte-level BPE tokenizer that closes the Brahmic compression gap at the 131K-vocabulary class while preserving the English, EU-language, and code compression of OpenAI's o200k_base. We construct it through a two-stage retrofit: (1) a script-prune crop that reduces 200,019 tokens to 131,072 by removing nine out-of-scope writing systems, and (2) a surgical retrofit of 2,372 corpus-dead vocabulary slots determined by linear-programming allocation across nine Brahmic Unicode blocks. The pre-tokenizer, decoder, and inherited merge rules are unchanged from o200k_base, making BrahmicTokenizer-131K a drop-in replacement at the tokenizer interface. On 27 million documents of public Indic pretraining text (2.84 billion words, 46.21 GB), BrahmicTokenizer-131K produces 26.7% fewer tokens than Mistral-Nemo Tekken / Sarvam-m at the same vocabulary budget, with per-language savings of 15.79% (Tamil) to 76.79% (Odia, a 4.31x compression ratio). The Odia advantage is mechanistically explained by Tekken/Sarvam-m containing zero Oriya-block tokens; our surgery added 725. On non-Indic content, BrahmicTokenizer-131K matches o200k_base's English fertility (1.235 vs 1.232 tokens/word) and beats Tekken/Sarvam-m by 4.0-14.2% on HumanEval, MBPP, and GSM8K. Across our 14-tokenizer benchmark, it is the only tokenizer simultaneously competitive on Brahmic, English, EU, code, and math at the 131K budget. Specialist tokenizers at other vocab classes (Sarvam-30B, Sarvam-1, MUTANT-Indic) achieve better Indic compression at the cost of non-Indic performance: Sarvam-1's English fertility is 15.9% worse and its code/math compression 26-33% worse than ours. We release the artifact under Apache 2.0 at https://huggingface.co/theschoolofai/BrahmicTokenizer-131K.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
20 of 32 references.
References · 20
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2025Gemma 3 Technical Reportcited by 1,824
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
- 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
- 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
- 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
- 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
- 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
8 more
- 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 102
- 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
- 2016Efficient softmax approximation for GPUscited by 67
- 2025SuperBPE: Space Travel for Language Modelscited by 63
- 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
- 2024Airavata: Introducing Hindi Instruction-tuned LLMcited by 32
- 2024SUTRA: Scalable Multilingual Language Model Architecturecited by 11
- 2025MUTANT: A Recipe for Multilingual Tokenizer Designcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.