mlsanityData to 7 Oct 2026

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

arXiv 2305.07185 ↗First posted 12 May 2023cs.LGcited by 164

Not on the map. Why?

Abstract (arXiv)

Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books. We proposed Megabyte, a multi-scale decoder architecture that enables end-to-end differentiable modeling of sequences of over one million bytes. Megabyte segments sequences into patches and uses a local submodel within patches and a global model between patches. This enables sub-quadratic self-attention, much larger feedforward layers for the same compute, and improved parallelism during decoding -- unlocking better performance at reduced cost for both training and generation. Extensive experiments show that Megabyte allows byte-level models to perform competitively with subword models on long context language modeling, achieve state-of-the-art density estimation on ImageNet, and model audio from raw files. Together, these results establish the viability of tokenization-free autoregressive sequence modeling at scale.

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). Builds on 44 of its 47 references that are in the map. First posted on arXiv 12 May 2023.

Builds on

44 of 47 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
32 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2017Mixed Precision Trainingcited by 1,135
  13. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  14. 2016Pixel Recurrent Neural Networkscited by 814
  15. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  16. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  17. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  18. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  19. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  20. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  21. 2018A Simple Method for Commonsense Reasoningcited by 223
  22. 2018Efficient Neural Audio Synthesiscited by 218
  23. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  24. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  25. 2022Block-Recurrent Transformerscited by 107
  26. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  27. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  28. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  29. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  30. 2022Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learningcited by 13
  31. 2022Subword-Delimited Downsampling for Better Character-Level Translationcited by 7
  32. 2023A Systematic Analysis of Vocabulary and BPE Settings for Optimal Fine-tuning of NMT: A Case Study of In-domain Translationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 49; 2025: 47; 2026: 48.

164 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  4. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2023Efficient Large Language Models: A Surveycited by 183
  7. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  8. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  9. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  10. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 112
  11. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  12. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
38 more
  1. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  2. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  3. 2025SuperBPE: Space Travel for Language Modelscited by 58
  4. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  5. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  6. 2025Fractal Generative Modelscited by 44
  7. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  8. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  9. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  10. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  11. 2024Logits of API-Protected LLMs Leak Proprietary Informationcited by 38
  12. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  13. 2024Cut Your Losses in Large-Vocabulary Language Modelscited by 35
  14. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  15. 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
  16. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  17. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  18. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  19. 2024Zero-Shot Tokenizer Transfercited by 30
  20. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  21. 2025Cognitive Memory in Large Language Modelscited by 28
  22. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  23. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  24. 2024B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memorycited by 26
  25. 2023TransFace++: Rethinking the Face Recognition Paradigm with a Focus on Accuracy, Efficiency, and Securitycited by 22
  26. 2024Beyond Language Models: Byte Models are Digital World Simulatorscited by 21
  27. 2024Where is the signal in tokenization space?cited by 21
  28. 2024Graph-Based Retriever Captures the Long Tail of Biomedical Knowledgecited by 20
  29. 2024Genomic Language Models: Opportunities and Challengescited by 20
  30. 2024Egalitarian Language Representation in Language Models: It All Begins with Tokenizerscited by 20
  31. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  32. 2025Scaling Embedding Layers in Language Modelscited by 19
  33. 2024Understanding and Mitigating Tokenization Bias in Language Modelscited by 18
  34. 2025FARMER: Flow AutoRegressive Transformer over Pixelscited by 17
  35. 2023FIT: Far-reaching Interleaved Transformerscited by 16
  36. 2023Bytes Are All You Need: Transformers Operating Directly On File Bytescited by 16
  37. 2024PIXAR: Auto-Regressive Language Modeling in Pixel Spacecited by 16
  38. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 16

50 most cited of 164 All 164 (JSON)