# Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier

arXiv: https://arxiv.org/abs/2504.00178 · HTML: https://mlsanity.com/paper/2504.00178/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 22 of 22)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://mlsanity.com/paper/2402.14903/) (2024)
- [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://mlsanity.com/paper/2310.08754/) (2023)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [The MiniPile Challenge for Data-Efficient Language Models](https://mlsanity.com/paper/2304.08442/) (2023)
- [Getting the most out of your tokenizer for pre-training and domain adaptation](https://mlsanity.com/paper/2402.01035/) (2024)
- [Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance](https://mlsanity.com/paper/2403.06265/) (2024)
- [Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling](https://mlsanity.com/paper/2501.16975/) (2025)
- [BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training](https://mlsanity.com/paper/2409.04599/) (2024)
- [Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings](https://mlsanity.com/paper/2109.03127/) (2021)
- [Egalitarian Language Representation in Language Models: It All Begins with Tokenizers](https://mlsanity.com/paper/2409.11501/) (2024)
- [Multi-word Tokenization for Sequence Compression](https://mlsanity.com/paper/2402.09949/) (2024)
- [Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal](https://mlsanity.com/paper/2404.17808/) (2024)
- [How Much is Enough? The Diminishing Returns of Tokenization Training Data](https://mlsanity.com/paper/2502.20273/) (2025)
- [Two Counterexamples to Tokenization and the Noiseless Channel](https://mlsanity.com/paper/2402.14614/) (2024)
- [Tokenization is Sensitive to Language Variation](https://mlsanity.com/paper/2502.15343/) (2025)
- [MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models](https://mlsanity.com/paper/2310.19531/) (2023)
- [An Analysis of BPE Vocabulary Trimming in Neural Machine Translation](https://mlsanity.com/paper/2404.00397/) (2024)
- [Don't Touch My Diacritics](https://mlsanity.com/paper/2410.24140/) (2024)

## Built on by (most cited first, 30 of 32)

- [Evaluating Morphological Alignment of Tokenizers in 70 Languages](https://mlsanity.com/paper/2507.06378/) (2025)
- [Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations](https://mlsanity.com/paper/2506.19004/) (2025)
- [BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization](https://mlsanity.com/paper/2505.24689/) (2025)
- [Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training](https://mlsanity.com/paper/2508.15390/) (2025)
- [Explaining and Mitigating Crosslingual Tokenizer Inequities](https://mlsanity.com/paper/2510.21909/) (2025)
- [Entropy-Driven Pre-Tokenization for Byte-Pair Encoding](https://mlsanity.com/paper/2506.15889/) (2025)
- [Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models](https://mlsanity.com/paper/2601.13260/) (2026)
- [MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment](https://mlsanity.com/paper/2606.27019/) (2026)
- [MUTANT: A Recipe for Multilingual Tokenizer Design](https://mlsanity.com/paper/2511.03237/) (2025)
- [TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior](https://mlsanity.com/paper/2512.20757/) (2025)
- [Tokenisation via Convex Relaxations](https://mlsanity.com/paper/2605.22821/) (2026)
- [SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance](https://mlsanity.com/paper/2508.11857/) (2025)
- [Proxy Compression for Language Modeling](https://mlsanity.com/paper/2602.04289/) (2026)
- [LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers](https://mlsanity.com/paper/2602.04706/) (2026)
- [TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar](https://mlsanity.com/paper/2510.14972/) (2025)
- [Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens](https://mlsanity.com/paper/2604.26355/) (2026)
- [Tokenization with Split Trees](https://mlsanity.com/paper/2605.22705/) (2026)
- [TokEval: A Tokenizer Evaluation Suite](https://mlsanity.com/paper/2608.18062/) (2026)
- [Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs](https://mlsanity.com/paper/2603.15773/) (2026)
- [Faster Superword Tokenization](https://mlsanity.com/paper/2604.05192/) (2026)
- [Tokenizer-Agnostic Engram Module](https://mlsanity.com/paper/2607.29065/) (2026)
- [HyperCLOVA X 32B Think](https://mlsanity.com/paper/2601.03286/) (2026)
- [Task-Centric Acceleration of Small-Language Models](https://mlsanity.com/paper/2602.24174/) (2026)
- [From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution](https://mlsanity.com/paper/2604.14053/) (2026)
- [Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)](https://mlsanity.com/paper/2605.22005/) (2026)
- [Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model](https://mlsanity.com/paper/2605.30080/) (2026)
- [Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES](https://mlsanity.com/paper/2607.05691/) (2026)
- [What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?](https://mlsanity.com/paper/2608.17325/) (2026)
- [To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs](https://mlsanity.com/paper/2609.15528/) (2026)
- [The Price of Token Boundaries: Compression Certificates and Prediction](https://mlsanity.com/paper/2609.35869/) (2026)

## Abstract (arXiv)

Pre-tokenization, the initial step in many modern tokenization pipelines, segments text into smaller units called pretokens, typically splitting on whitespace and punctuation. While this process encourages having full, individual words as tokens, it introduces a fundamental limitation in most tokenization algorithms such as Byte Pair Encoding (BPE). Specifically, pre-tokenization causes the distribution of tokens in a corpus to heavily skew towards common, full-length words. This skewed distribution limits the benefits of expanding to larger vocabularies, since the additional tokens appear with progressively lower counts. To overcome this barrier, we propose BoundlessBPE, a modified BPE algorithm that relaxes the pretoken boundary constraint. Our approach selectively merges two complete pretokens into a larger unit we term a superword. Superwords are not necessarily semantically cohesive. For example, the pretokens " of" and " the" might be combined to form the superword " of the". This merging strategy results in a substantially more uniform distribution of tokens across a corpus than standard BPE, and compresses text more effectively, with up to a 15% increase in bytes per token.