mlsanityData to 7 Oct 2026

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation

arXiv 2604.27263 ↗First posted 29 Apr 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understood. In this work, we decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline. We formulate and test hypotheses across various dimensions, including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. By simulating these effects in a byte-level setting, we refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Specifically, our experiments highlight the critical role of increased training throughput and the integration of subword boundaries as either explicit priors or inductive biases.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

27 of 37 references.

References · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  6. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  7. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  8. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  9. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  10. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  11. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  12. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 137
15 more
  1. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  2. 2024Tokenization Is More Than Compressioncited by 87
  3. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 80
  4. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  5. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  6. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  7. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  8. 2023A Formal Perspective on Byte-Pair Encodingcited by 35
  9. 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 23
  10. 2022N-Grammer: Augmenting Transformers with latent n-gramscited by 13
  11. 2026Scaling Embeddings Outperforms Scaling Experts in Language Modelscited by 13
  12. 2025The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Modelscited by 11
  13. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  14. 2025Exploiting Vocabulary Frequency Imbalance in Language Model Pre-trainingcited by 7
  15. 2026Proxy Compression for Language Modelingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Efficient Pre-Training with Token Superpositioncited by 1
  2. 2026Rethinking Contextualization by Reinterpreting Attention Head Channelscited by 0