# Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation

arXiv: https://arxiv.org/abs/2510.09947 · HTML: https://mlsanity.com/paper/2510.09947/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Builds on (most cited first, 17 of 17)

- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://mlsanity.com/paper/2310.08754/) (2023)
- [Multilingual Instruction Tuning With Just a Pinch of Multilinguality](https://mlsanity.com/paper/2401.01854/) (2024)
- [Getting the most out of your tokenizer for pre-training and domain adaptation](https://mlsanity.com/paper/2402.01035/) (2024)
- [Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance](https://mlsanity.com/paper/2403.06265/) (2024)
- [Tokenization and the Noiseless Channel](https://mlsanity.com/paper/2306.16842/) (2023)
- [Analyzing Cognitive Plausibility of Subword Tokenization](https://mlsanity.com/paper/2310.13348/) (2023)
- [Unobserved Local Structures Make Compositional Generalization Hard](https://mlsanity.com/paper/2201.05899/) (2022)
- [MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies](https://mlsanity.com/paper/2502.00894/) (2025)
- [Evaluating Morphological Alignment of Tokenizers in 70 Languages](https://mlsanity.com/paper/2507.06378/) (2025)
- [Lexically Grounded Subword Segmentation](https://mlsanity.com/paper/2406.13560/) (2024)
- [Beyond Text Compression: Evaluating Tokenizers Across Scales](https://mlsanity.com/paper/2506.03101/) (2025)
- [Sub-Character Tokenization for Chinese Pretrained Language Models](https://mlsanity.com/paper/2106.00400/) (2021)

## Built on by (most cited first, 3 of 3)

- [Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models](https://mlsanity.com/paper/2601.13260/) (2026)
- [How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipeline](https://mlsanity.com/paper/2604.04204/) (2026)
- [Lost in Translation: Do LVLM Judges Generalize Across Languages?](https://mlsanity.com/paper/2604.19405/) (2026)

## Abstract (arXiv)

Tokenization is a crucial but under-evaluated step in large language models (LLMs). The standard metric, fertility (the average number of tokens per word), captures compression efficiency but obscures how vocabularies are allocated across languages and domains. We analyze six widely used tokenizers across seven languages and two domains, finding stable fertility for English, high fertility for Chinese, and little domain sensitivity. To address fertility's blind spots, we propose the Single Token Retention Rate (STRR), which measures the proportion of words preserved as single tokens. STRR reveals systematic prioritization of English, strong support for Chinese, and fragmentation in Hindi, offering an interpretable view of cross-lingual fairness. Our results show that STRR complements fertility and provides practical guidance for designing more equitable multilingual tokenizers.