mlsanityData to 14 Oct 2026

Compute Optimal Tokenization

arXiv 2605.01188 ↗First posted 2 May 2026cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this work, we systematically investigate how the information granularity of tokens, controlled by the compression rate (i.e., average bytes of text per token), affects scaling trends. We train 988 latent tokenized models (BLT) ranging from 50M to 7B parameters that enable setting the desired compression rate. This flexibility allows us to study the role of compression rate well beyond 4.57 bytes per token obtained with a popular BPE tokenizer. Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes, not in tokens as commonly perceived (Kaplan et al., 2020; Hoffmann et al., 2022). Furthermore, we discover that the optimal compression rate differs from the one obtained with BPE and decreases with compute. These findings generalize to both latent and subword tokenization, as well as to languages other than English, guiding language model developers on tokenization scheme selection for maximal compute efficiency.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 2 May 2026.

Builds on

39 of 46 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
27 more
  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  2. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  3. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  4. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  5. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  6. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  7. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  8. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  9. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  10. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  11. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  12. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  13. 2024Tokenization Is More Than Compressioncited by 89
  14. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  15. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  16. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  17. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 62
  18. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  19. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  20. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 34
  21. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  22. 2024Scaling Laws for Multilingual Language Modelscited by 23
  23. 2025Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Modelscited by 22
  24. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  25. 2025From Bytes to Ideas: Language Modeling with Autoregressive U-Netscited by 15
  26. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  27. 2025Scaling Laws for Code: Every Programming Language Matterscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Abra: Scaling Diffusion Image Trainingcited by 1
  2. 2026Lifecycle-Optimal Tokenization: Vocabulary Size as a Deployment-Regime-Dependent Infrastructure Parametercited by 0
  3. 2026Removable and Irreducible: A Token-Cost Ledger for the Multilingual Tokenization Taxcited by 0
  4. 2026Byte Language Models: Scaling, Emergent Abstractions, and Information Allocationcited by 0