mlsanityData to 14 Oct 2026

Easy to anticipate, hard to compute: boundary dependence finds the computed outputs that entropy patching misses

arXiv 2610.11790 ↗First posted 8 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Byte-level language models such as the Byte Latent Transformer (BLT) group bytes into patches and run their large global model once per patch. BLT starts a patch where a small model's next-byte entropy is high, so global compute goes where the next byte is hard to predict. We show that this rule has a systematic blind spot: positions whose type is predictable but whose value must be computed, such as the number after "=" in a worked math solution. Under tight patch budgets, entropy-triggered layouts skip these positions, and accuracy on them collapses. In Meta's BLT-1B with patch starts on 10% of bytes, the entropy rule puts a patch start at 16% of the computed results in GSM8K solutions and gets 19.0% of them exactly right; a boundary after each "=" at the same patch count gets 51.8%, and entropy combined with a label-free boundary-dependence signal gets 67.1% (default layout at 26% of bytes: 76.8%). The gap survives adapting BLT-1B to the budget with low-rank fine-tuning (32.9% vs 72.7%, three runs per rule, paired p < 1e-200) and grows with model size in byte models trained from scratch at a 10% budget: at 1M, 12M and 50M parameters, boundary dependence beats entropy on final answers by -1.6, +10.1 and +19.8 points, and at 50M it gets 35.9% of computed results against 13.9% (3 seeds each). BLT's entropy-jump rule helps neither target at 50M. The entropy trigger of Scratchpad Patching is likewise indistinguishable from random scratchpads on final answers (5.6% vs 5.9%, 5 seeds), while answer-start scratchpads give 38.1%. The effect is specific to computed values: copies and lookups gain little, and values the model cannot compute gain nothing. Boundary dependence, the rise in the model's own loss when a patch start is removed, measured per two-byte context, finds these positions without labels: combined with entropy it beats the hand-written rule on computed results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 17 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

17 of 17 references.

References · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  4. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  5. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  6. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  7. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  8. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  9. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  10. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 34
  11. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 28
  12. 2025From Bytes to Ideas: Language Modeling with Autoregressive U-Netscited by 15
5 more
  1. 2026Fast Byte Latent Transformercited by 3
  2. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  3. 2025ByteSpan: Information-Driven Subword Tokenisationcited by 1
  4. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  5. 2026EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMscited by 1

Built on by

Not cited yet.