mlsanityData to 14 Oct 2026

Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models

arXiv 2601.13260 ↗First posted 19 Jan 2026cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Tokenization underlies every large language model, yet it remains an under-theorized and inconsistently designed component. Common subword approaches such as Byte Pair Encoding (BPE) offer scalability but often misalign with linguistic structure, amplify bias, and waste capacity across languages and domains. This paper reframes tokenization as a core modeling decision rather than a preprocessing step. We argue for a context-aware framework that integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. Standardized evaluation and transparent reporting are essential to make tokenization choices accountable and comparable. Treating tokenization as a core design problem, not a technical afterthought, can yield language technologies that are fairer, more efficient, and more adaptable.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 66 of its 92 references that are in the map. First posted on arXiv 19 Jan 2026.

Builds on

66 of 92 references.

References · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  7. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  8. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  9. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  10. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  11. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  12. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
38 more
  1. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  2. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  3. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  4. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  5. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  6. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  7. 2024Tokenization Is More Than Compressioncited by 89
  8. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
  9. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  10. 2025SuperBPE: Space Travel for Language Modelscited by 63
  11. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  12. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  13. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  14. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 44
  15. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  16. 2021Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domainscited by 36
  17. 2023A Formal Perspective on Byte-Pair Encodingcited by 36
  18. 2023Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspectivecited by 34
  19. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  20. 2024Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?cited by 33
  21. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  22. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 32
  23. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  24. 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 31
  25. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  26. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 28
  27. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 25
  28. 2024Exploring Design Choices for Building Language-Specific LLMscited by 22
  29. 2025Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Modelscited by 22
  30. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  31. 2024Egalitarian Language Representation in Language Models: It All Begins with Tokenizerscited by 20
  32. 2024Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removalcited by 16
  33. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 16
  34. 2021Joint Optimization of Tokenization and Downstream Modelcited by 15
  35. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  36. 2018Optimizing Segmentation Granularity for Neural Machine Translationcited by 14
  37. 2024Lexically Grounded Subword Segmentationcited by 13
  38. 2025Tokenization is Sensitive to Language Variationcited by 13

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipelinecited by 3
  2. 2026Domain Adapted Large Language Models for Additive Manufacturingcited by 0
  3. 2026The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrievalcited by 0
  4. 2026Where to cut, how deep: BPE and Unigram-LM on chemistry SMILEScited by 0
  5. 2026What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?cited by 0
  6. 2026When Tokenization is Secretly Output Supervisioncited by 0