# Two Counterexamples to Tokenization and the Noiseless Channel

arXiv: https://arxiv.org/abs/2402.14614 · HTML: https://mlsanity.com/paper/2402.14614/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 6 of 6)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation](https://mlsanity.com/paper/2005.06606/) (2020)
- [A Formal Perspective on Byte-Pair Encoding](https://mlsanity.com/paper/2306.16837/) (2023)
- [Joint Optimization of Tokenization and Downstream Model](https://mlsanity.com/paper/2105.12410/) (2021)

## Built on by (most cited first, 13 of 13)

- [Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier](https://mlsanity.com/paper/2504.00178/) (2025)
- [Evaluating Morphological Alignment of Tokenizers in 70 Languages](https://mlsanity.com/paper/2507.06378/) (2025)
- [Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment](https://mlsanity.com/paper/2508.08424/) (2025)
- [MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment](https://mlsanity.com/paper/2606.27019/) (2026)
- [Tokenisation via Convex Relaxations](https://mlsanity.com/paper/2605.22821/) (2026)
- [CharBench: Evaluating the Role of Tokenization in Character-Level Tasks](https://mlsanity.com/paper/2508.02591/) (2025)
- [UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8](https://mlsanity.com/paper/2511.05578/) (2025)
- [TokEval: A Tokenizer Evaluation Suite](https://mlsanity.com/paper/2608.18062/) (2026)
- [Length-MAX Tokenizer for Language Models](https://mlsanity.com/paper/2511.20849/) (2025)
- [Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES](https://mlsanity.com/paper/2607.05691/) (2026)
- [MoirfEolas and Cr'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology](https://mlsanity.com/paper/2609.05022/) (2026)
- [Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining](https://mlsanity.com/paper/2609.21362/) (2026)
- [The Price of Token Boundaries: Compression Certificates and Prediction](https://mlsanity.com/paper/2609.35869/) (2026)

## Abstract (arXiv)

In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), R'enyi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest R'enyi efficiency of the unigram distribution should be chosen. The R'enyi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of this metric is not perfect, and the authors note there are additional qualities of a good tokenization scheme that R'enyi efficiency alone cannot capture. We describe two variants of BPE tokenization which can arbitrarily increase R'enyi efficiency while decreasing the downstream model performance. These counterexamples expose cases where R'enyi efficiency fails as an intrinsic tokenization metric and thus give insight for building more accurate predictors.