# Patch where the text is new: a compression view of where byte-level models should spend compute

arXiv: https://arxiv.org/abs/2610.22100 · HTML: https://mlsanity.com/paper/2610.22100/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Tokenization and byte-level models, one of 8 directions in LLM pre-training data and scaling laws.
- Builds on 19 of its 21 references that are in the map.
- First posted on arXiv 15 Oct 2026.

## Placed in

- LLM pre-training data and scaling laws / [Tokenization and byte-level models](https://mlsanity.com/pretrain/tokenization-and-byte-level-models/)

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Byte Latent Transformer: Patches Scale Better Than Tokens](https://mlsanity.com/paper/2412.09871/) (2024)
- [OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text](https://mlsanity.com/paper/2310.06786/) (2023)
- [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022)
- [Dynamic Chunking for End-to-End Hierarchical Sequence Modeling](https://mlsanity.com/paper/2507.07955/) (2025)
- [SpaceByte: Towards Deleting Tokenization from Large Language Modeling](https://mlsanity.com/paper/2404.14408/) (2024)
- [MrT5: Dynamic Token Merging for Efficient Byte-level Language Models](https://mlsanity.com/paper/2410.20771/) (2024)
- [From Bytes to Ideas: Language Modeling with Autoregressive U-Nets](https://mlsanity.com/paper/2506.14761/) (2025)
- [zip2zip: Inference-Time Adaptive Tokenization via Online Compression](https://mlsanity.com/paper/2506.01084/) (2025)
- [Fast Byte Latent Transformer](https://mlsanity.com/paper/2605.08044/) (2026)
- [ByteSpan: Information-Driven Subword Tokenisation](https://mlsanity.com/paper/2506.18639/) (2025)
- [SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models](https://mlsanity.com/paper/2601.22805/) (2026)
- [ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer](https://mlsanity.com/paper/2603.03583/) (2026)
- [Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models](https://mlsanity.com/paper/2605.09630/) (2026)
- [Disentangling Language Modeling and Boundaries](https://mlsanity.com/paper/2608.03599/) (2026)
- [EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs](https://mlsanity.com/paper/2608.06398/) (2026)
- [Easy to anticipate, hard to compute: boundary dependence finds the computed outputs that entropy patching misses](https://mlsanity.com/paper/2610.11790/) (2026)

## Abstract (arXiv)

Byte-level language models such as the Byte Latent Transformer (BLT) run their large global model once per patch and start a patch where a small model's next-byte entropy is high. This rule rarely starts a patch at in-line computed results, such as the number after "=" in a worked solution, and accuracy on them collapses under tight patch budgets. We propose novelty patching, motivated by LZ77-style compression: start a patch where the upcoming bytes are unlikely to repeat the recent context. The rule is a lookup table over the previous two bytes, fitted in seconds on raw text with no labels, reference model or losses. In BLT-style models trained from scratch on a 1.9 GB math corpus at a matched ~11% patch rate (3 seeds each), it has no clear effect at 1M parameters, raises final-answer accuracy from 47% to 72% at 12M, and at 50M raises computed-result accuracy from 21% to 57% and final-answer accuracy from 60% to 90%; a table fitted on the raw pretraining corpus keeps the gain. Outside math the advantage shrinks: novelty lowers bits per byte on prose and ties entropy on source code and on synthetic execution traces. An analysis of where boundaries fall explains the pattern. Computed results have ordinary surprisal for the patcher's model, so surprisal-seeking rules pass over them: entropy concentrates patch starts on high-surprisal bytes most strongly of all rules (boundary enrichment 1.61) and is the worst on computed results, while the best rules sit near chance enrichment. Novelty works where new content begins at segment starts close to the computation, as in written math; a learned chunker (H-Net) finds such structure on its own, in math and in code. We also report what did not work: typed uncertainty as a training rule, switching rules after training, and a context-aware novelty predictor.