mlsanityData to 14 Oct 2026

The Token Tax: Systematic Bias in Multilingual Tokenization

arXiv 2509.05486 ↗First posted 5 Sep 2025cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Tokenization inefficiency imposes structural disadvantages on morphologically complex, low-resource languages, inflating compute resources and depressing accuracy. We evaluate 10 large language models (LLMs) on AfriMMLU (9,000 MCQA items; 5 subjects; 16 African languages) and show that fertility (tokens/word) reliably predicts accuracy. Higher fertility consistently predicts lower accuracy across all models and subjects. We further find that reasoning models (DeepSeek, o1) consistently outperform non-reasoning peers across high and low resource languages in the AfriMMLU dataset, narrowing accuracy gaps observed in prior generations. Finally, translating token inflation to economics, a doubling in tokens results in quadrupled training cost and time, underscoring the token tax faced by many languages. These results motivate morphologically aware tokenization, fair pricing, and multilingual benchmarks for equitable natural language processing (NLP).

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 13 of its 19 references that are in the map. First posted on arXiv 5 Sep 2025.

Builds on

13 of 19 references.

References · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  3. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  4. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  5. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  6. 2022On The Computational Complexity of Self-Attentioncited by 139
  7. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  8. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  9. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  10. 2025Where Are We? Evaluating LLM Performance on African Languagescited by 10
  11. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  12. 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
1 more
  1. 2022Local Byte Fusion for Neural Machine Translationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  2. 2026TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languagescited by 3
  3. 2026Tokenization with Split Treescited by 2
  4. 2026Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabetcited by 2
  5. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  6. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  7. 2026AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkanicited by 1
  8. 2026Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translationcited by 0
  9. 2026Modular Monolingual Adaptation using Pretrained Language Modelscited by 0
  10. 2026Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Languagecited by 0
  11. 2026The Role of Prompt Language and Translation-Theory-Driven Prompts in Large Language Models: A Case Study on Spanish-Chinese Journalistic Translationcited by 0
  12. 2026Removable and Irreducible: A Token-Cost Ledger for the Multilingual Tokenization Taxcited by 0
2 more
  1. 2026What Does Post-Training Change in Multilingual Reasoning?cited by 0
  2. 2026The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers, and a French-Optimized Prototypecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
  2. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languages
  3. 2025EfficientXLang: Towards Improving Token Efficiency Through Cross-Lingual Reasoning
  4. 2024Better & Faster Large Language Models via Multi-token Prediction
  5. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs
  6. 2025Beyond Literal Token Overlap: Token Alignability for Multilinguality
  7. 2026Say Anything but This: When Tokenizer Betrays Reasoning in LLMs
  8. 2025TASE: Token Awareness and Structured Evaluation for Multilingual Language Models