mlsanityData to 14 Oct 2026

The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty

arXiv 2605.24718 ↗First posted 23 May 2026cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Tokenizer fertility the number of tokens per word imposes a hidden cost on non-English NLP. We measure fertility for ten foundation models across 25 European languages on parallel text, producing the first controlled tokenizer tax map for the continent. The tax spans 2.5x from English (1.2 tokens/word) to Greek/Maltese (~3.1), following a clear hierarchy: Romance (1.5-1.7), Germanic (1.7-1.9), Slavic (2.2-2.5), Uralic/Baltic (2.7-3.0). Ukrainian (2.7) pays 15-18% more than cognate Slavic languages, reflecting underrepresentation in pre-training data. Fertility rankings are domain-invariant across three text registers (rho > 0.97). A subword analysis reveals that high-fertility tokenizers fragment morphological boundaries rather than preserving them. Cross-lingual few-shot evaluation on four Slavic languages shows that few-shot effects are model-intrinsic, not language-dependent. We release all measurements as a public dataset.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 23 May 2026.

Builds on

19 of 30 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  8. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  9. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  10. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  11. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  12. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
7 more
  1. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  2. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  3. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  4. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  5. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  6. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  7. 2026Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Studycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoningcited by 1
  2. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditionscited by 1
  3. 2026The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMscited by 0
  4. 2026SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responsescited by 0