mlsanityData to 7 Oct 2026

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

arXiv 2512.20757 ↗First posted 23 Dec 2025cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we release fourteen pre-trained models that use different off-the-shelf tokenizers but are otherwise identical, using the same architecture, dataset, training budget, and initialization. We also release a multilingual robustness benchmark that measures model performance under real-world perturbations in English, Chinese, Farsi, Italian, and Turkish, curated by native annotators. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 52 of its 75 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

52 of 75 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
38 more
  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  10. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  12. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  13. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  14. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  15. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  16. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 137
  17. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
  18. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  19. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  20. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  21. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  22. 2024Tokenization Is More Than Compressioncited by 87
  23. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  24. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  25. 2025SuperBPE: Space Travel for Language Modelscited by 58
  26. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  27. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  28. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  29. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 31
  30. 2025The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Textcited by 25
  31. 2024Tokenization Falling Short: On Subword Robustness in Large Language Modelscited by 18
  32. 2024Training LLMs over Neurally Compressed Textcited by 16
  33. 2024Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensemblescited by 15
  34. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  35. 2022A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learningcited by 12
  36. 2023Effects of sub-word segmentation on performance of transformer language modelscited by 12
  37. 2025Adversarial Tokenizationcited by 11
  38. 2024Token Alignment via Character Matching for Subword Completioncited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 5
  2. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  3. 2026Stochasticity in Tokenisation Improves Robustnesscited by 1
  4. 2026Language-Specific Effects of Tokenizer Choice in Multilingual Language Modelscited by 0