mlsanityData to 14 Oct 2026

Beyond Text Compression: Evaluating Tokenizers Across Scales

arXiv 2506.03101 ↗First posted 3 Jun 2025cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

The choice of tokenizer can profoundly impact language model performance, yet accessible and reliable evaluations of tokenizer quality remain an open challenge. Inspired by scaling consistency, we show that smaller models can accurately predict significant differences in tokenizer impact on larger models at a fraction of the compute cost. By systematically evaluating both English-centric and multilingual tokenizers, we find that tokenizer choice has negligible effects on tasks in English but results in consistent performance differences in multilingual settings. We propose new intrinsic tokenizer metrics inspired by Zipf's law that correlate more strongly with downstream performance than text compression when modeling unseen languages. By combining several metrics to capture multiple aspects of tokenizer behavior, we develop a reliable framework for intrinsic tokenizer evaluations. Our work offers a more efficient path to informed tokenizer selection in future language model development.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 63 of its 103 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

63 of 103 references.

References · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  11. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  13. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  14. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  15. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  16. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  17. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  18. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  19. 2023The Falcon Series of Open Language Modelscited by 535
  20. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  21. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  22. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  23. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  24. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  25. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  26. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  27. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  28. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  29. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  30. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  31. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  32. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  33. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  34. 2024Coercing LLMs to do and reveal (almost) anythingcited by 89
  35. 2024Tokenization Is More Than Compressioncited by 89
  36. 2022Inverse scaling can become U-shapedcited by 76
  37. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 75
  38. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  2. 2025Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluationcited by 3
  3. 2026Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabetcited by 2
  4. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  5. 2026Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertilitycited by 2
  6. 2026Writing-System-Level Tokenizer Adaptation for Byte-Level BPEcited by 1
  7. 2025Languages are Modalities: Cross-Lingual Alignment via Encoder Injectioncited by 0
  8. 2026How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizerscited by 0
  9. 2026Enabling Cloud-Level Accuracy in Edge AI through IoT Data Preprocessingcited by 0
  10. 2026From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inferencecited by 0
  11. 2026When Tokenization is Secretly Output Supervisioncited by 0