mlsanityData to 14 Oct 2026

Byte Latent Transformer: Patches Scale Better Than Tokens

arXiv 2412.09871 ↗First posted 13 Dec 2024cs.CLcited by 146

Not on the map. Why?

Abstract (arXiv)

We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). Builds on 1 of its 51 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

1 of 51 references.

References · 1

  1. 2019Bridging the Gap for Tokenizer-Free Language Modelscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 64; 2026: 81.

146 papers.

Sort:

By area:

  1. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  2. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 91
  3. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 86
  4. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  5. 2025SuperBPE: Space Travel for Language Modelscited by 63
  6. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 44
  7. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 41
  8. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  9. 2025Scaling Embedding Layers in Language Modelscited by 20
  10. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  11. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 17
  12. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
38 more
  1. 2025From Bytes to Ideas: Language Modeling with Autoregressive U-Netscited by 15
  2. 2026Scaling Embeddings Outperforms Scaling Experts in Language Modelscited by 15
  3. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 14
  4. 2025False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Modelscited by 11
  5. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  6. 2024Tokenisation is NP-Completecited by 10
  7. 2025Learning to Skip the Middle Layers of Transformerscited by 10
  8. 2025Don't Throw Away Your Pretrained Modelcited by 10
  9. 2025Continuous Autoregressive Language Modelscited by 10
  10. 2025Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Taskscited by 9
  11. 2025See the Text: From Tokenization to Visual Readingcited by 9
  12. 2025Fast and Expressive Multi-Byte Prediction with Probabilistic Circuitscited by 9
  13. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 9
  14. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  15. 2025Revisiting associative recall in modern recurrent modelscited by 8
  16. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 8
  17. 2026Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalizationcited by 8
  18. 2025Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformerscited by 7
  19. 2025Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modelingcited by 7
  20. 2025StochasTok: Improving Fine-Grained Subword Understanding in LLMscited by 7
  21. 2025E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Modelscited by 7
  22. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  23. 2025Byte-token Enhanced Language Models for Temporal Point Processes Analysiscited by 6
  24. 2025Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generationcited by 6
  25. 2025FreeMesh: Boosting Mesh Generation with Coordinates Mergingcited by 6
  26. 2025Entropy-Driven Pre-Tokenization for Byte-Pair Encodingcited by 6
  27. 2025Context-level Language Modeling by Learning Predictive Context Embeddingscited by 6
  28. 2025Model-Aware Tokenizer Transfercited by 6
  29. 2025A Partition Cover Approach to Tokenizationcited by 5
  30. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 5
  31. 2025Latent Speech-Text Transformercited by 5
  32. 2026Debugging code world modelscited by 5
  33. 2025PixelWorld: How Far Are We from Perceiving Everything as Pixels?cited by 4
  34. 2025EXECUTE: A Multilingual Benchmark for LLM Token Understandingcited by 4
  35. 2025TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behaviorcited by 4
  36. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 4
  37. 2026Compute Optimal Tokenizationcited by 4
  38. 2025Scaling LLM Pre-training with Vocabulary Curriculumcited by 3

50 most cited of 146 All 146 (JSON)