# SpaceByte: Towards Deleting Tokenization from Large Language Modeling

arXiv: https://arxiv.org/abs/2404.14408 · HTML: https://mlsanity.com/paper/2404.14408/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 60 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [A Simple Method for Commonsense Reasoning](https://mlsanity.com/paper/1806.02847/) (2018)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers](https://mlsanity.com/paper/2305.07185/) (2023)

## Built on by (most cited first, 30 of 34)

- [Dynamic Chunking for End-to-End Hierarchical Sequence Modeling](https://mlsanity.com/paper/2507.07955/) (2025)
- [Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models](https://mlsanity.com/paper/2405.05417/) (2024)
- [MrT5: Dynamic Token Merging for Efficient Byte-level Language Models](https://mlsanity.com/paper/2410.20771/) (2024)
- [From Bytes to Ideas: Language Modeling with Autoregressive U-Nets](https://mlsanity.com/paper/2506.14761/) (2025)
- [Learning to Skip the Middle Layers of Transformers](https://mlsanity.com/paper/2506.21103/) (2025)
- [Bolmo: Byteifying the Next Generation of Language Models](https://mlsanity.com/paper/2512.15586/) (2025)
- [zip2zip: Inference-Time Adaptive Tokenization via Online Compression](https://mlsanity.com/paper/2506.01084/) (2025)
- [Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization](https://mlsanity.com/paper/2601.23174/) (2026)
- [Compute Optimal Tokenization](https://mlsanity.com/paper/2605.01188/) (2026)
- [Membership Inference Attacks on Tokenizers of Large Language Models](https://mlsanity.com/paper/2510.05699/) (2025)
- [Controllably Efficient Language Models](https://mlsanity.com/paper/2511.05313/) (2025)
- [ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocation](https://mlsanity.com/paper/2601.21420/) (2026)
- [Proxy Compression for Language Modeling](https://mlsanity.com/paper/2602.04289/) (2026)
- [TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting](https://mlsanity.com/paper/2603.11352/) (2026)
- [Fast Byte Latent Transformer](https://mlsanity.com/paper/2605.08044/) (2026)
- [MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging](https://mlsanity.com/paper/2511.14806/) (2025)
- [Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet](https://mlsanity.com/paper/2606.20993/) (2026)
- [HATSolver: Learning Groebner Bases with Hierarchical Attention Transformers](https://mlsanity.com/paper/2512.14722/) (2025)
- [Say Anything but This: When Tokenizer Betrays Reasoning in LLMs](https://mlsanity.com/paper/2601.14658/) (2026)
- [ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer](https://mlsanity.com/paper/2603.03583/) (2026)
- [HoloByte: Continuous Hyperspherical Distillation for Tokenizer-Free Modeling](https://mlsanity.com/paper/2603.16917/) (2026)
- [Dynamic Tokenization via Reinforcement Patching: End-to-end Training and Zero-shot Transfer](https://mlsanity.com/paper/2603.26097/) (2026)
- [Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models](https://mlsanity.com/paper/2605.09630/) (2026)
- [Scaling Particle Collision Data Analysis](https://mlsanity.com/paper/2412.00129/) (2024)
- [DNACHUNKER: Learnable Tokenization for DNA Language Models](https://mlsanity.com/paper/2601.03019/) (2026)
- [Distilling Token-Trained Models into Byte-Level Models](https://mlsanity.com/paper/2602.01007/) (2026)
- [You Can Learn Tokenization End-to-End with Reinforcement Learning](https://mlsanity.com/paper/2602.13940/) (2026)
- [GeneZip: Region-Aware Compression for Long Context DNA Modeling](https://mlsanity.com/paper/2602.17739/) (2026)
- [LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling](https://mlsanity.com/paper/2606.04552/) (2026)
- [Toppling the Hierarchy in Byte-level Language Modeling](https://mlsanity.com/paper/2609.00463/) (2026)

## Abstract (arXiv)

Tokenization is widely used in large language models because it significantly improves performance. However, tokenization imposes several disadvantages, such as performance biases, increased adversarial vulnerability, decreased character-level modeling performance, and increased modeling complexity. To address these disadvantages without sacrificing performance, we propose SpaceByte, a novel byte-level decoder architecture that closes the performance gap between byte-level and subword autoregressive language modeling. SpaceByte consists of a byte-level Transformer model, but with extra larger transformer blocks inserted in the middle of the layers. We find that performance is significantly improved by applying these larger blocks only after certain bytes, such as space characters, which typically denote word boundaries. Our experiments show that for a fixed training and inference compute budget, SpaceByte outperforms other byte-level architectures and roughly matches the performance of tokenized Transformer architectures.