# Large Byte Model: Teaching Language Models About Compiled Code

arXiv: https://arxiv.org/abs/2606.02834 · HTML: https://mlsanity.com/paper/2606.02834/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://mlsanity.com/paper/2309.14509/) (2023)
- [MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers](https://mlsanity.com/paper/2305.07185/) (2023)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [PalmTree: Learning an Assembly Language Model for Instruction Embedding](https://mlsanity.com/paper/2103.03809/) (2021)
- [Cut Your Losses in Large-Vocabulary Language Models](https://mlsanity.com/paper/2411.09009/) (2024)
- [Classifying Sequences of Extreme Length with Constant Memory Applied to Malware Detection](https://mlsanity.com/paper/2012.09390/) (2020)
- [Beyond Language Models: Byte Models are Digital World Simulators](https://mlsanity.com/paper/2402.19155/) (2024)
- [Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models](https://mlsanity.com/paper/2402.14714/) (2024)
- [Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models](https://mlsanity.com/paper/2312.09601/) (2023)
- [Transformers for End-to-End InfoSec Tasks: A Feasibility Study](https://mlsanity.com/paper/2212.02666/) (2022)
- [Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents](https://mlsanity.com/paper/2602.02164/) (2026)

## Abstract (arXiv)

Malware analysis starts with the raw bytes of an executable program, and tools to "lift" these to higher-level representations, such as assembly, are expensive and subject to error. Large Language Models (LLMs) cannot process raw byte representations and answer questions about them. To this end, we present the first byte-native LLM. Based on a vocabulary expansion technique using a bespoke byte tokenizer, such a model is capable of responding to complex questions about malware binaries, with accuracies ranging from 69% for malware family classification to 98% for architecture classification. Our findings indicate that providing domain knowledge during training is essential for this application -- off-the-shelf models lack both accuracy and insight. We've deployed this emerging solution to a limited number of analysts to gather feedback for further improvements.