mlsanityData to 7 Oct 2026

MambaByte: Token-free Selective State Space Model

arXiv 2401.13660 ↗First posted 24 Jan 2024cs.CL cs.LGcited by 58

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

also in 1 more
Abstract (arXiv)

Token-free language models learn directly from raw bytes and remove the inductive bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences. In this setting, standard autoregressive Transformers scale poorly as the effective memory required grows with sequence length. The recent development of the Mamba state space model (SSM) offers an appealing alternative approach with a fixed-sized memory state and efficient decoding. We propose MambaByte, a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences. In terms of modeling, we show MambaByte to be competitive with, and even to outperform, state-of-the-art subword Transformers on language modeling tasks while maintaining the benefits of token-free language models, such as robustness to noise. In terms of efficiency, we develop an adaptation of speculative decoding with tokenized drafting and byte-level verification. This results in a 2.6× inference speedup to the standard MambaByte implementation, showing similar decoding efficiency as the subword Mamba. These findings establish the viability of SSMs in enabling token-free language modeling.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 54 of its 67 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

54 of 67 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  3. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  4. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections24 Jun
  5. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  6. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
36 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  5. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  6. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  7. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  8. 2022Long Range Language Modeling via Gated State Spacescited by 265
  9. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  10. 2018A Simple Method for Commonsense Reasoningcited by 223
  11. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  12. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  13. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  14. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  15. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  16. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  17. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  18. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  19. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  20. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  21. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  22. 2023Online Speculative Decodingcited by 109
  23. 2022Block-Recurrent Transformerscited by 107
  24. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  25. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  26. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  27. 2023Diffusion Models Without Attentioncited by 75
  28. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  29. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  30. 2023Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecturecited by 62
  31. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  32. 2022Pretraining Without Attentioncited by 51
  33. 2022ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Modelscited by 25
  34. 2019Bridging the Gap for Tokenizer-Free Language Modelscited by 21
  35. 2020Character-Level Translation with Self-attentioncited by 12
  36. 2018Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Modelcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 15; 2026: 16.

58 papers.

Sort:

By area:

  1. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  2. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  3. 2024The Hidden Attention of Mamba Modelscited by 88
  4. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  5. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  6. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  7. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  8. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  9. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  10. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  11. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  12. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 25
38 more
  1. 2024Matten: Video Generation with Mamba-Attentioncited by 24
  2. 2024MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoderscited by 22
  3. 2024PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learningcited by 21
  4. 2024Exploring the Capability of Mamba in Speech Applicationscited by 19
  5. 2025Scaling Embedding Layers in Language Modelscited by 19
  6. 2024Selective Attention: Enhancing Transformer through Principled Context Controlcited by 16
  7. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 16
  8. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 14
  9. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  10. 2025Uncertainty Distillation: Teaching Language Models to Express Semantic Confidencecited by 13
  11. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  12. 2024MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoderscited by 10
  13. 2024ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Modelscited by 9
  14. 2025Safe-Sora: Safe Text-to-Video Generation via Graphical Watermarkingcited by 9
  15. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  16. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  17. 2024Mamba-ST: State Space Model for Efficient Style Transfercited by 5
  18. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognitioncited by 5
  19. 2026Proxy Compression for Language Modelingcited by 3
  20. 2026A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Modelscited by 3
  21. 2024Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generationcited by 2
  22. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  23. 2025Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mambacited by 2
  24. 2025Differential Mambacited by 2
  25. 2026Fast Byte Latent Transformercited by 2
  26. 2026Prefix Sliding for efficient test-time scalingcited by 2
  27. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mambacited by 1
  28. 2024PixelBytes: Catching Unified Representation for Multimodal Generationcited by 1
  29. 2024Exploring contextual modeling with linear complexity for point cloud segmentationcited by 1
  30. 2026MambaFormer: Token-Level Guided Routing Mixture-of-Experts for Accurate and Efficient Clinical Assistancecited by 1
  31. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  32. 2026On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashingcited by 1
  33. 2024PixelBytes: Catching Unified Embedding for Multimodal Generationcited by 0
  34. 2025MambaStyle: Efficient StyleGAN Inversion for Real Image Editing with State-Space Modelscited by 0
  35. 2025S2M2ECG: Spatio-temporal bi-directional State Space Model Enabled Multi-branch Mamba for ECGcited by 0
  36. 2025Mamba-driven multi-perspective structural understanding for molecular ground-state conformation predictioncited by 0
  37. 2025Arcee: Differentiable Recurrent State Chain for Generative Vision Modeling with Mamba SSMscited by 0
  38. 2026Distilling Token-Trained Models into Byte-Level Modelscited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Models
  2. 2025Mamba Drafters for Speculative Decoding
  3. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  4. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  5. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  7. 2024SSSD: Simply-Scalable Speculative Decoding
  8. 2024SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens