mlsanityData to 14 Oct 2026

Explaining and Mitigating Crosslingual Tokenizer Inequities

arXiv 2510.21909 ↗First posted 24 Oct 2025cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called token premiums. Having high token premiums leads to less throughput during training and increases costs at inference. In this paper, we show that even after controlling for dataset size, vocabulary size, and data content, monolingual tokenizers exhibit a wide range of token premiums across languages. To understand the cross-linguistic differences that cause these token premiums, we train a suite of approximately 7,000 comparable monolingual tokenizers for 97 languages, manipulating tokenization algorithm, vocabulary size, and dataset size. We measure token premiums and test for a relationship between factors such as data similarity (between tokenizer training and evaluation), vocabulary size, and pre-tokenization. We also investigate the role of language-specific features such as writing system and word length. We find that similarity between training and test data does not impact token premiums, but vocabulary size and pre-tokenization do. While simply increasing vocabulary size does not lead to reduced token premium effects, we can determine an ``optimal'' vocabulary size for each language to achieve significantly reduced token premium effects. We also train superword tokenizers which allow merges over whitespaces, and we find that they both reduce token premium effects and improve compression overall. Thus, intervening on the vocabulary size or the pre-tokenizer significantly reduces crosslingual token premium effects.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

34 of 52 references.

References · 34

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  7. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  8. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  9. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  10. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  11. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  12. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
22 more
  1. 2023Language Modeling Is Compressioncited by 225
  2. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  3. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  4. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  5. 2024Tokenization Is More Than Compressioncited by 89
  6. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  7. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  8. 2025SuperBPE: Space Travel for Language Modelscited by 63
  9. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 62
  10. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  11. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  12. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  13. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 32
  14. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  15. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  16. 2024A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languagescited by 25
  17. 2019Bridging the Gap for Tokenizer-Free Language Modelscited by 21
  18. 2024Egalitarian Language Representation in Language Models: It All Begins with Tokenizerscited by 20
  19. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  20. 2025Tokenization is Sensitive to Language Variationcited by 13
  21. 2025BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenizationcited by 12
  22. 2024The Case for Co-Designing Model Architectures with Hardwarecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  2. 2026Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamicscited by 3
  3. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  4. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  5. 2026An In-Vitro Study on Cross-Lingual Generalization in Language Modelscited by 0
  6. 2026Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Propositioncited by 0
  7. 2026To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMscited by 0