mlsanityData to 14 Oct 2026

Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation

arXiv 2510.09947 ↗First posted 11 Oct 2025cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Tokenization is a crucial but under-evaluated step in large language models (LLMs). The standard metric, fertility (the average number of tokens per word), captures compression efficiency but obscures how vocabularies are allocated across languages and domains. We analyze six widely used tokenizers across seven languages and two domains, finding stable fertility for English, high fertility for Chinese, and little domain sensitivity. To address fertility's blind spots, we propose the Single Token Retention Rate (STRR), which measures the proportion of words preserved as single tokens. STRR reveals systematic prioritization of English, strong support for Chinese, and fragmentation in Hindi, offering an interpretable view of cross-lingual fairness. Our results show that STRR complements fertility and provides practical guidance for designing more equitable multilingual tokenizers.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

17 of 26 references.

References · 17

  1. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  2. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  3. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  4. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  5. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  6. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  7. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  8. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  9. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  10. 2023Tokenization and the Noiseless Channelcited by 59
  11. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  12. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
5 more
  1. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 25
  2. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  3. 2024Lexically Grounded Subword Segmentationcited by 13
  4. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  5. 2021Sub-Character Tokenization for Chinese Pretrained Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 6
  2. 2026How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipelinecited by 3
  3. 2026Lost in Translation: Do LVLM Judges Generalize Across Languages?cited by 0