mlsanityData to 19 Oct 2026

Tokenization and byte-level models

51 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: tokenization, subword, language, tokenizer, byte-level

Key facts. Tokenization and byte-level models: 51 papers on arXiv since 2020 (data through 19 Oct 2026). 19 of its 51 mapped papers (37%) cite “CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation” (2021, arXiv 2103.06874). 4 new papers 1 Oct–19 Oct.

What’s new

1 Oct – 19 Oct 2026

  1. Earlier · 4
  2. Patch where the text is new: a compression view of where byte-level models should spend compute also fits: Pre-training corpora and quality filtering
  3. In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B
  4. More Than Words: Compositional Tokenization for Efficient Language Models
  5. Counting and Min-Cost Encoding for Tokenization in Large Language Models

Foundations

Most cited by papers here

  1. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation2021cited by 19 here
  2. ByT5: Towards a token-free future with pre-trained byte-to-byte models2021cited by 19 here
  3. Byte Pair Encoding is Suboptimal for Language Model Pretraining2020cited by 12 here
  4. Charformer: Fast Character Transformers via Gradient-based Subword Tokenization2021cited by 9 here
  5. Byte Latent Transformer: Patches Scale Better Than Tokens2024cited by 7 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 12 here
  2. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 19 here
  3. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 19 here
  4. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 9 here
  5. 2023CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Codecited by 2 here
  6. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 3 here
18 more it builds on, not in this chain
  1. 2020Scaling Laws for Neural Language Modelscited by 3 here
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1 here
  3. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 2 here
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 5 here
  5. 2022Training Compute-Optimal Large Language Modelscited by 3 here
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 2 here
  7. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 4 here
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 3 here
  9. 2024Toward a Theory of Tokenization in LLMscited by 2 here
  10. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 3 here
  11. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 2 here
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 7 here
  13. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 4 here
  14. 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 2 here
  15. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 2 here
  16. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 2 here
  17. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 2 here
  18. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 7 here

The 24 most cited of 48.

Browse

30 of 51, ranked 6–35

  1. 6Analyzing Cognitive Plausibility of Subword Tokenization20234
  2. 7Bolmo: Byteifying the Next Generation of Language Models20253
  3. 8Toward a Theory of Tokenization in LLMs20242
  4. 9BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training20242
  5. 10MrT5: Dynamic Token Merging for Efficient Byte-level Language Models20242
  6. 11CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code20232
  7. 12Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation20261
  8. 13H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages20251
  9. 14Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training20251
  10. 15Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)20241
All 30
  1. 16ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models20221
  2. 17Sub-Character Tokenization for Chinese Pretrained Language Models20211
  3. 18Efficient Pre-Training with Token Superposition20260
  4. 19SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization20260
  5. 20Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model20260
  6. 21moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT20260
  7. 22The Functionalizer: Lossless Functional Decomposition for Subword Tokenization20260
  8. 23The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages20260
  9. 24Disentangling Language Modeling and Boundaries20260
  10. 25Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages20260
  11. 26Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?20260
  12. 27Toppling the Hierarchy in Byte-level Language Modeling20260
  13. 28Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining20260
  14. 29MDL-Calibrated Significance-Gain Pair Encoding: Replication-Aware Automatic Stopping for Subword Tokenization20260
  15. 30Token-free Models for Sarcasm Detection20250
  16. 31Comparative analysis of subword tokenization approaches for Indian languages20250
  17. 32ByteSpan: Information-Driven Subword Tokenisation20250
  18. 33FLEXITOKENS: Flexible Tokenization for Evolving Language Models20250
  19. 34Innovative tokenisation of structured data for LLM training20250
  20. 35The Art of Breaking Words: Rethinking Multilingual Tokenizer Design20250