mlsanityData to 7 Oct 2026

Beyond Language Models: Byte Models are Digital World Simulators

arXiv 2402.19155 ↗First posted 29 Feb 2024cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Traditional deep learning often overlooks bytes, the basic units of the digital world, where all forms of information and operations are encoded and manipulated in binary format. Inspired by the success of next token prediction in natural language processing, we introduce bGPT, a model with next byte prediction to simulate the digital world. bGPT matches specialized models in performance across various modalities, including text, audio, and images, and offers new possibilities for predicting, simulating, and diagnosing algorithm or hardware behaviour. It has almost flawlessly replicated the process of converting symbolic music data, achieving a low error rate of 0.0011 bits per byte in converting ABC notation to MIDI format. In addition, bGPT demonstrates exceptional capabilities in simulating CPU behaviour, with an accuracy exceeding 99.99% in executing various operations. Leveraging next byte prediction, models like bGPT can directly learn from vast binary data, effectively simulating the intricate patterns of the digital world.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 43 of its 54 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

43 of 54 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
31 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2018Deep contextualized word representationscited by 2,285
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2022High Fidelity Neural Audio Compressioncited by 673
  10. 2021AST: Audio Spectrogram Transformercited by 634
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  13. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  14. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  15. 2023Simple and Controllable Music Generationcited by 480
  16. 2023Language Modeling Is Compressioncited by 225
  17. 2018Music Transformercited by 207
  18. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  19. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  20. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  21. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  22. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  23. 2017Malware Detection by Eating a Whole EXEcited by 114
  24. 2021Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphscited by 109
  25. 2021MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Trainingcited by 89
  26. 2022Tranception: protein fitness prediction with autoregressive transformers and inference-time retrievalcited by 78
  27. 2021Improved Fine-Tuning by Better Leveraging Pre-Training Datacited by 36
  28. 2023TunesFormer: Forming Irish Tunes with Control Codes by Bar Patchingcited by 20
  29. 2021Training Multilingual Pre-trained Language Model with Byte-level Subwordscited by 16
  30. 2023Bytes Are All You Need: Transformers Operating Directly On File Bytescited by 16
  31. 2023CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrievalcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 8; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024Lossless data compression by large modelscited by 20
  2. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  3. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7
  4. 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compressioncited by 7
  5. 2024ByteNet: Rethinking Multimedia File Fragment Classification through Visual Perspectivescited by 5
  6. 2024Compressed-Language Models for Understanding Compressed File Formats: a JPEG Explorationcited by 4
  7. 2024JPEG-LM: LLMs as Image Generators with Canonical Codec Representationscited by 3
  8. 2025Scaling LLM Pre-training with Vocabulary Curriculumcited by 3
  9. 2026Proxy Compression for Language Modelingcited by 3
  10. 2024FuzzCoder: Byte-level Fuzzing Test via Large Language Modelcited by 2
  11. 2024Exploring Tokenization Methods for Multitrack Sheet Music Generationcited by 2
  12. 2025MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Mergingcited by 2
9 more
  1. 2026Bitstream Action Recognition is Byte Modelingcited by 2
  2. 2025GViT: Representing Images as Gaussians for Visual Recognitioncited by 1
  3. 2024Scaling Particle Collision Data Analysiscited by 0
  4. 2025Malware Detection in Docker Containers: An Image is Worth a Thousand Logscited by 0
  5. 2025StackPilot: Autonomous Function Agents for Scalable and Environment-Free Code Executioncited by 0
  6. 2025Transformers from Compressed Representationscited by 0
  7. 2026Byte-level generative predictions for forensics multimedia carvingcited by 0
  8. 2026Large Byte Model: Teaching Language Models About Compiled Codecited by 0
  9. 2026ByteAction: Byte-space Action Recognition Foundation Modelcited by 0