# Tokenization and byte-level models

Area: [LLM pre-training data and scaling laws](https://mlsanity.com/pretrain/index.md). HTML: https://mlsanity.com/pretrain/tokenization-and-byte-level-models/

## Key facts

- Tokenization and byte-level models: 51 papers on arXiv since 2020 (data through 19 Oct 2026).
- 19 of its 51 mapped papers (37%) cite “CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation” (2021, arXiv 2103.06874).
- 19 of its 51 mapped papers (37%) cite “ByT5: Towards a token-free future with pre-trained byte-to-byte models” (2021, arXiv 2105.13626).
- 4 new papers 1 Oct–19 Oct.

## Most cited here

- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021), cited by 19 papers here
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021), cited by 19 papers here
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020), cited by 12 papers here
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021), cited by 9 papers here
- [Byte Latent Transformer: Patches Scale Better Than Tokens](https://mlsanity.com/paper/2412.09871/) (2024), cited by 7 papers here
- [Analyzing Cognitive Plausibility of Subword Tokenization](https://mlsanity.com/paper/2310.13348/) (2023), cited by 4 papers here
- [Bolmo: Byteifying the Next Generation of Language Models](https://mlsanity.com/paper/2512.15586/) (2025), cited by 3 papers here
- [CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code](https://mlsanity.com/paper/2308.00683/) (2023), cited by 2 papers here
- [Toward a Theory of Tokenization in LLMs](https://mlsanity.com/paper/2404.08335/) (2024), cited by 2 papers here
- [BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training](https://mlsanity.com/paper/2409.04599/) (2024), cited by 2 papers here

## New (1 Oct–19 Oct)

- [Patch where the text is new: a compression view of where byte-level models should spend compute](https://mlsanity.com/paper/2610.22100/) (2026-10-15)
- [In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B](https://mlsanity.com/paper/2610.15373/) (2026-10-11)
- [More Than Words: Compositional Tokenization for Efficient Language Models](https://mlsanity.com/paper/2610.05597/) (2026-10-04)
- [Counting and Min-Cost Encoding for Tokenization in Large Language Models](https://mlsanity.com/paper/2610.01127/) (2026-10-01)