# Beyond Language Models: Byte Models are Digital World Simulators

arXiv: https://arxiv.org/abs/2402.19155 · HTML: https://mlsanity.com/paper/2402.19155/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 54 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)

## Built on by (most cited first, 22 of 22)

- [Lossless data compression by large models](https://mlsanity.com/paper/2407.07723/) (2024)
- [Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters](https://mlsanity.com/paper/2408.04093/) (2024)
- [Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery](https://mlsanity.com/paper/2505.16477/) (2025)
- [Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression](https://mlsanity.com/paper/2511.08066/) (2025)
- [ByteNet: Rethinking Multimedia File Fragment Classification through Visual Perspectives](https://mlsanity.com/paper/2410.20855/) (2024)
- [Compressed-Language Models for Understanding Compressed File Formats: a JPEG Exploration](https://mlsanity.com/paper/2405.17146/) (2024)
- [JPEG-LM: LLMs as Image Generators with Canonical Codec Representations](https://mlsanity.com/paper/2408.08459/) (2024)
- [Exploring Tokenization Methods for Multitrack Sheet Music Generation](https://mlsanity.com/paper/2410.17584/) (2024)
- [Scaling LLM Pre-training with Vocabulary Curriculum](https://mlsanity.com/paper/2502.17910/) (2025)
- [Proxy Compression for Language Modeling](https://mlsanity.com/paper/2602.04289/) (2026)
- [FuzzCoder: Byte-level Fuzzing Test via Large Language Model](https://mlsanity.com/paper/2409.01944/) (2024)
- [MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging](https://mlsanity.com/paper/2511.14806/) (2025)
- [Bitstream Action Recognition is Byte Modeling](https://mlsanity.com/paper/2608.15695/) (2026)
- [GViT: Representing Images as Gaussians for Visual Recognition](https://mlsanity.com/paper/2506.23532/) (2025)
- [Scaling Particle Collision Data Analysis](https://mlsanity.com/paper/2412.00129/) (2024)
- [Malware Detection in Docker Containers: An Image is Worth a Thousand Logs](https://mlsanity.com/paper/2504.03238/) (2025)
- [StackPilot: Autonomous Function Agents for Scalable and Environment-Free Code Execution](https://mlsanity.com/paper/2508.11665/) (2025)
- [Transformers from Compressed Representations](https://mlsanity.com/paper/2510.23665/) (2025)
- [Byte-level generative predictions for forensics multimedia carving](https://mlsanity.com/paper/2604.11010/) (2026)
- [Large Byte Model: Teaching Language Models About Compiled Code](https://mlsanity.com/paper/2606.02834/) (2026)
- [ByteAction: Byte-space Action Recognition Foundation Model](https://mlsanity.com/paper/2608.22760/) (2026)
- [Image Bitstream Fine-grained Understanding for Privacy-Friendly AIoT](https://mlsanity.com/paper/2610.08414/) (2026)

## Abstract (arXiv)

Traditional deep learning often overlooks bytes, the basic units of the digital world, where all forms of information and operations are encoded and manipulated in binary format. Inspired by the success of next token prediction in natural language processing, we introduce bGPT, a model with next byte prediction to simulate the digital world. bGPT matches specialized models in performance across various modalities, including text, audio, and images, and offers new possibilities for predicting, simulating, and diagnosing algorithm or hardware behaviour. It has almost flawlessly replicated the process of converting symbolic music data, achieving a low error rate of 0.0011 bits per byte in converting ABC notation to MIDI format. In addition, bGPT demonstrates exceptional capabilities in simulating CPU behaviour, with an accuracy exceeding 99.99% in executing various operations. Leveraging next byte prediction, models like bGPT can directly learn from vast binary data, effectively simulating the intricate patterns of the digital world.