mlsanityData to 7 Oct 2026

SpaceByte: Towards Deleting Tokenization from Large Language Modeling

arXiv 2404.14408 ↗First posted 22 Apr 2024cs.CL cs.AI cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Tokenization is widely used in large language models because it significantly improves performance. However, tokenization imposes several disadvantages, such as performance biases, increased adversarial vulnerability, decreased character-level modeling performance, and increased modeling complexity. To address these disadvantages without sacrificing performance, we propose SpaceByte, a novel byte-level decoder architecture that closes the performance gap between byte-level and subword autoregressive language modeling. SpaceByte consists of a byte-level Transformer model, but with extra larger transformer blocks inserted in the middle of the layers. We find that performance is significantly improved by applying these larger blocks only after certain bytes, such as space characters, which typically denote word boundaries. Our experiments show that for a fixed training and inference compute budget, SpaceByte outperforms other byte-level architectures and roughly matches the performance of tokenized Transformer architectures.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 52 of its 60 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

52 of 60 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
38 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  3. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  4. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  5. 2021Dynamic Neural Networks: A Surveycited by 405
  6. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  7. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  8. 2016Using the Output Embedding to Improve Language Modelscited by 266
  9. 2020Query-Key Normalization for Transformerscited by 266
  10. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  11. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  12. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  13. 2018A Simple Method for Commonsense Reasoningcited by 223
  14. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  15. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  16. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  17. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  18. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  19. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  20. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  21. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  22. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  23. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  24. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  25. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  26. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  27. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  28. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  29. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  30. 2024Training LLMs over Neurally Compressed Textcited by 16
  31. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 14
  32. 2020Multi-scale Transformer Language Modelscited by 12
  33. 2022A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learningcited by 12
  34. 2022Subword-Delimited Downsampling for Better Character-Level Translationcited by 7
  35. 2023Learn Your Tokens: Word-Pooled Tokenization for Language Modelingcited by 7
  36. 2023Toucan: Token-Aware Character Level Language Modelingcited by 7
  37. 2023What is the best recipe for character-level encoder-only modelling?cited by 6
  38. 2022Local Byte Fusion for Neural Machine Translationcited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 19.

31 papers.

Sort:

By area:

  1. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  2. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  3. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  4. 2025From Bytes to Ideas: Language Modeling with Autoregressive U-Netscited by 15
  5. 2025Learning to Skip the Middle Layers of Transformerscited by 8
  6. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  7. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 3
  8. 2025Membership Inference Attacks on Tokenizers of Large Language Modelscited by 3
  9. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 3
  10. 2026Proxy Compression for Language Modelingcited by 3
  11. 2026TimeSqueeze: Dynamic Patching for Efficient Time Series Forecastingcited by 3
  12. 2025Controllably Efficient Language Modelscited by 2
19 more
  1. 2025MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Mergingcited by 2
  2. 2026ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocationcited by 2
  3. 2026Compute Optimal Tokenizationcited by 2
  4. 2026Fast Byte Latent Transformercited by 2
  5. 2026Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabetcited by 2
  6. 2025HATSolver: Learning Groebner Bases with Hierarchical Attention Transformerscited by 1
  7. 2026Say Anything but This: When Tokenizer Betrays Reasoning in LLMscited by 1
  8. 2026HoloByte: Continuous Hyperspherical Distillation for Tokenizer-Free Modelingcited by 1
  9. 2026Dynamic Tokenization via Reinforcement Patching: End-to-end Training and Zero-shot Transfercited by 1
  10. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  11. 2024Scaling Particle Collision Data Analysiscited by 0
  12. 2026DNACHUNKER: Learnable Tokenization for DNA Language Modelscited by 0
  13. 2026Distilling Token-Trained Models into Byte-Level Modelscited by 0
  14. 2026You Can Learn Tokenization End-to-End with Reinforcement Learningcited by 0
  15. 2026GeneZip: Region-Aware Compression for Long Context DNA Modelingcited by 0
  16. 2026ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizercited by 0
  17. 2026LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modelingcited by 0
  18. 2026Toppling the Hierarchy in Byte-level Language Modelingcited by 0
  19. 2026Easy to anticipate, hard to compute: boundary dependence finds the computed outputs that entropy patching missescited by 0