mlsanityData to 14 Oct 2026

Tokenization and the Noiseless Channel

arXiv 2306.16842 ↗First posted 29 Jun 2023cs.CL cs.IT math.ITcited by 59

Not on the map. Why?

Abstract (arXiv)

Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others. We propose that good tokenizers lead to efficient channel usage, where the channel is the means by which some input is conveyed to the model and efficiency can be quantified in information-theoretic terms as the ratio of the Shannon entropy to the maximum possible entropy of the token distribution. Yet, an optimal encoding according to Shannon entropy assigns extremely long codes to low-frequency tokens and very short codes to high-frequency tokens. Defining efficiency in terms of R'enyi entropy, on the other hand, penalizes distributions with either very high or very low-frequency tokens. In machine translation, we find that across multiple tokenizers, the R'enyi entropy with α = 2.5 has a very strong correlation with textscBleu: 0.78 in comparison to just -0.32 for compressed length.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 16 of its 47 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

16 of 47 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  8. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  9. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  10. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  11. 2014On Using Very Large Target Vocabulary for Neural Machine Translationcited by 80
  12. 2016Character-based Neural Machine Translationcited by 33
4 more
  1. 2023What changes when you randomly choose BPE merge operations? Not muchcited by 10
  2. 2016Vocabulary Manipulation for Neural Machine Translationcited by 5
  3. 2016Vocabulary Selection Strategies for Neural Machine Translationcited by 4
  4. 2022Checks and Strategies for Enabling Code-Switched Machine Translationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 17; 2025: 21; 2026: 18.

59 papers.

Sort:

  1. 2024Tokenization Is More Than Compressioncited by 89
  2. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  3. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 48
  4. 2024How to Compute the Probability of a Wordcited by 36
  5. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  6. 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 31
  7. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 23
  8. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 22
  9. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  10. 2024Subobject-level Image Tokenizationcited by 21
  11. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  12. 2024Understanding and Mitigating Tokenization Bias in Language Modelscited by 18
38 more
  1. 2025Causal Estimation of Tokenisation Biascited by 16
  2. 2024Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensemblescited by 15
  3. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  4. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  5. 2025Confounding Factors in Relating Model Performance to Morphologycited by 14
  6. 2024Lexically Grounded Subword Segmentationcited by 13
  7. 2025Tokenization is Sensitive to Language Variationcited by 13
  8. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  9. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  10. 2024Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challengecited by 8
  11. 2024An Analysis of BPE Vocabulary Trimming in Neural Machine Translationcited by 7
  12. 2025MorphTok: Morphologically Grounded Tokenization for Indian Languagescited by 7
  13. 2025Exploiting Vocabulary Frequency Imbalance in Language Model Pre-trainingcited by 7
  14. 2025Splintering Nonconcatenative Languages for Better Tokenizationcited by 6
  15. 2023Headless Language Models: Learning without Predicting with Contrastive Weight Tyingcited by 5
  16. 2024The Impact of Token Granularity on the Predictive Power of Language Model Surprisalcited by 5
  17. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  18. 2023Scaling Experiments in Self-Supervised Cross-Table Representation Learningcited by 4
  19. 2024Distributional Properties of Subword Regularizationcited by 4
  20. 2025MUTANT: A Recipe for Multilingual Tokenizer Designcited by 4
  21. 2025Bit-level BPE: Below the byte boundarycited by 3
  22. 2025Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluationcited by 3
  23. 2025UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8cited by 3
  24. 2026Proxy Compression for Language Modelingcited by 3
  25. 2025Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insightscited by 2
  26. 2026Syllabic Agglutinative Tokenizations for Indonesian LLM: A Study from Gasing Literacy Learning Systemcited by 2
  27. 2026Tokenization with Split Treescited by 2
  28. 2025Uncovering inequalities in new knowledge learning by large language models across different languagescited by 1
  29. 2025Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performancecited by 1
  30. 2025ByteSpan: Information-Driven Subword Tokenisationcited by 1
  31. 2025Length-MAX Tokenizer for Language Modelscited by 1
  32. 2026Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Featurescited by 1
  33. 2026Faster Superword Tokenizationcited by 1
  34. 2026Effective Context in Transformers: An Analysis of Fragmentation and Tokenizationcited by 1
  35. 2024Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealingcited by 0
  36. 2024Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenizationcited by 0
  37. 2026Task-Centric Acceleration of Small-Language Modelscited by 0
  38. 2026Emergent retokenization symmetry in large language models: phenomenology and applicationscited by 0

50 most cited of 59 All 59 (JSON)