mlsanityData to 7 Oct 2026

The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models

arXiv 2602.11174 ↗First posted 19 Jan 2026cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Pretrained multilingual language models are often assumed to be script-agnostic, yet their tokenizers can impose systematic costs on certain writing systems. We quantify this script tax by comparing two orthographic variants with identical linguistic content. Across mBERT and XLM-R, the higher-fragmentation orthography shows a ~3.4x increase in fertility (6.73-6.85 vs. 2.10-2.35 tokens/word), leading to a 16.5x inference slowdown (0.23 vs. 3.8 sentences/second) on identical hardware. Using bits per character (BPC) to avoid the "NLL paradox" from subword fragmentation, we find a substantial increase in information cost: +19.7% for mBERT (8.06->9.65) and +47.1% for XLM-R (12.19->17.94). A round-trip conversion check (CER_rt=0.31) suggests these gaps reflect orthography-conditioned processing rather than mapping noise. Our results highlight tokenization as a key source of inequity in multilingual NLP and motivate script-aware tokenization and pretraining.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 19 Jan 2026.

Builds on

16 of 23 references.

References · 16

  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  2. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  3. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  4. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  5. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  6. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  7. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  8. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 61
  9. 2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
  10. 2023Comparing Biases and the Impact of Multilingual Training across Multiple Languagescited by 36
  11. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  12. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
4 more
  1. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  2. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 14
  3. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  4. 2022A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inferencecited by 2
  2. 2026Tokenization with Split Treescited by 2
  3. 2026Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Servingcited by 1