mlsanityData to 7 Oct 2026

ByteSpan: Information-Driven Subword Tokenisation

arXiv 2506.18639 ↗First posted 23 Jun 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Recent dynamic tokenisation methods operate directly on bytes and pool their latent representations into patches. This bears similarities to computational models of word segmentation that determine lexical boundaries using spikes in an autoregressive model's prediction error. Inspired by this connection, we explore whether grouping predictable bytes - rather than pooling their representations - can yield a useful fixed subword vocabulary. We propose a new information-driven subword tokeniser, ByteSpan, that uses an external byte-level LM during training to identify contiguous predictable byte sequences and group them into subwords. Experiments show that ByteSpan yields efficient vocabularies with higher morphological alignment scores than BPE for English. Multilingual experiments show similar compression and R'enyi efficiency for 25 languages.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 45 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

24 of 45 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  6. 2021Scaling Vision Transformerscited by 993
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  8. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  9. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  10. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  11. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  12. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
12 more
  1. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  2. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  3. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 75
  4. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  5. 2023Tokenization and the Noiseless Channelcited by 58
  6. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  7. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  8. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  9. 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19
  10. 2024Tokenization Falling Short: On Subword Robustness in Large Language Modelscited by 18
  11. 2023CompoundPiece: Evaluating and Improving Decompounding Performance of Language Modelscited by 7
  12. 2022Lost in Space Markingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Easy to anticipate, hard to compute: boundary dependence finds the computed outputs that entropy patching missescited by 0