# Byte Latent Transformer: Patches Scale Better Than Tokens

arXiv: https://arxiv.org/abs/2412.09871 · HTML: https://mlsanity.com/paper/2412.09871/

## Key facts

- Cited by 146 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 51 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Builds on (most cited first, 1 of 1)

- [Bridging the Gap for Tokenizer-Free Language Models](https://mlsanity.com/paper/1908.10322/) (2019)

## Built on by (most cited first, 30 of 146)

- [A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond](https://mlsanity.com/paper/2503.21614/) (2025)
- [Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning](https://mlsanity.com/paper/2502.03275/) (2025)
- [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://mlsanity.com/paper/2601.07372/) (2026)
- [Dynamic Chunking for End-to-End Hierarchical Sequence Modeling](https://mlsanity.com/paper/2507.07955/) (2025)
- [SuperBPE: Space Travel for Language Models](https://mlsanity.com/paper/2503.13423/) (2025)
- [Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling](https://mlsanity.com/paper/2501.16975/) (2025)
- [Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching](https://mlsanity.com/paper/2503.20083/) (2025)
- [Energy-Based Transformers are Scalable Learners and Thinkers](https://mlsanity.com/paper/2507.02092/) (2025)
- [Scaling Embedding Layers in Language Models](https://mlsanity.com/paper/2502.01637/) (2025)
- [Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization](https://mlsanity.com/paper/2508.04796/) (2025)
- [Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations](https://mlsanity.com/paper/2506.19004/) (2025)
- [AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation](https://mlsanity.com/paper/2503.19693/) (2025)
- [From Bytes to Ideas: Language Modeling with Autoregressive U-Nets](https://mlsanity.com/paper/2506.14761/) (2025)
- [Scaling Embeddings Outperforms Scaling Experts in Language Models](https://mlsanity.com/paper/2601.21204/) (2026)
- [Hybrid Architectures for Language Models: Systematic Analysis and Design Insights](https://mlsanity.com/paper/2510.04800/) (2025)
- [False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models](https://mlsanity.com/paper/2509.18750/) (2025)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Tokenisation is NP-Complete](https://mlsanity.com/paper/2412.15210/) (2024)
- [Learning to Skip the Middle Layers of Transformers](https://mlsanity.com/paper/2506.21103/) (2025)
- [Don't Throw Away Your Pretrained Model](https://mlsanity.com/paper/2510.09913/) (2025)
- [Continuous Autoregressive Language Models](https://mlsanity.com/paper/2510.27688/) (2025)
- [Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks](https://mlsanity.com/paper/2509.20045/) (2025)
- [See the Text: From Tokenization to Visual Reading](https://mlsanity.com/paper/2510.18840/) (2025)
- [Fast and Expressive Multi-Byte Prediction with Probabilistic Circuits](https://mlsanity.com/paper/2511.11346/) (2025)
- [Bolmo: Byteifying the Next Generation of Language Models](https://mlsanity.com/paper/2512.15586/) (2025)
- [FLEXITOKENS: Flexible Tokenization for Evolving Language Models](https://mlsanity.com/paper/2507.12720/) (2025)
- [Revisiting associative recall in modern recurrent models](https://mlsanity.com/paper/2508.19029/) (2025)
- [Cross-Tokenizer LLM Distillation through a Byte-Level Interface](https://mlsanity.com/paper/2604.07466/) (2026)
- [Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization](https://mlsanity.com/paper/2607.22334/) (2026)
- [Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers](https://mlsanity.com/paper/2501.02393/) (2025)

## Abstract (arXiv)

We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.