# ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

arXiv: https://arxiv.org/abs/2603.03583 · HTML: https://mlsanity.com/paper/2603.03583/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 3 Mar 2026.

## Builds on (most cited first, 30 of 30)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers](https://mlsanity.com/paper/2305.07185/) (2023)
- [Byte Latent Transformer: Patches Scale Better Than Tokens](https://mlsanity.com/paper/2412.09871/) (2024)
- [Hierarchical Transformers Are More Efficient Language Models](https://mlsanity.com/paper/2110.13711/) (2021)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022)
- [Dynamic Chunking for End-to-End Hierarchical Sequence Modeling](https://mlsanity.com/paper/2507.07955/) (2025)
- [Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT](https://mlsanity.com/paper/2003.04985/) (2020)
- [SuperBPE: Space Travel for Language Models](https://mlsanity.com/paper/2503.13423/) (2025)
- [MambaByte: Token-free Selective State Space Model](https://mlsanity.com/paper/2401.13660/) (2024)
- [A Formal Perspective on Byte-Pair Encoding](https://mlsanity.com/paper/2306.16837/) (2023)
- [SpaceByte: Towards Deleting Tokenization from Large Language Modeling](https://mlsanity.com/paper/2404.14408/) (2024)
- [Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers](https://mlsanity.com/paper/2512.17351/) (2025)
- [Counting Ability of Large Language Models and Impact of Tokenization](https://mlsanity.com/paper/2410.19730/) (2024)
- [CUTE: Measuring LLMs' Understanding of Their Tokens](https://mlsanity.com/paper/2409.15452/) (2024)
- [MrT5: Dynamic Token Merging for Efficient Byte-level Language Models](https://mlsanity.com/paper/2410.20771/) (2024)
- [From Bytes to Ideas: Language Modeling with Autoregressive U-Nets](https://mlsanity.com/paper/2506.14761/) (2025)

## Built on by (most cited first, 1 of 1)

- [Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation](https://mlsanity.com/paper/2610.05978/) (2026)

## Abstract (arXiv)

Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce ByteFlow Net, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries while preserving a static computation graph via Top-K selection. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.