mlsanityData to 14 Oct 2026

From Bytes to Ideas: Language Modeling with Autoregressive U-Nets

arXiv 2506.14761 ↗First posted 17 Jun 2025cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Tokenization imposes a fixed granularity on the input text, freezing how a language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and similar schemes split text once, build a static vocabulary, and leave the model stuck with that choice. We relax this rigidity by introducing an autoregressive U-Net that learns to embed its own tokens as it trains. The network reads raw bytes, pools them into words, then pairs of words, then up to 4 words, giving it a multi-scale view of the sequence. At deeper stages, the model must predict further into the future -- anticipating the next few words rather than the next byte -- so deeper stages focus on broader semantic patterns while earlier stages handle fine details. When carefully tuning and controlling pretraining compute, shallow hierarchies tie strong BPE baselines, and deeper hierarchies have a promising trend. Because tokenization now lives inside the model, the same system can handle character-level tasks and carry knowledge across low-resource languages.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

23 of 32 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  8. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  9. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  10. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  11. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  12. 2024Language models scale reliably with over-training and on downstream taskscited by 113
11 more
  1. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  2. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 102
  3. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  4. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  5. 2024Memory Layers at Scalecited by 48
  6. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 34
  7. 2024CUTE: Measuring LLMs' Understanding of Their Tokenscited by 31
  8. 2025Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Modelscited by 22
  9. 2024Training LLMs over Neurally Compressed Textcited by 17
  10. 2024T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddingscited by 10
  11. 2024ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Modelcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 11.

15 papers.

Sort:

By area:

  1. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  2. 2025Learning to Skip the Middle Layers of Transformerscited by 10
  3. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 8
  4. 2025Latent Speech-Text Transformercited by 5
  5. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 4
  6. 2026Compute Optimal Tokenizationcited by 4
  7. 2026ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocationcited by 3
  8. 2026Proxy Compression for Language Modelingcited by 3
  9. 2026Efficient Pre-Training with Token Superpositioncited by 2
  10. 2025HATSolver: Learning Groebner Bases with Hierarchical Attention Transformerscited by 1
  11. 2026ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizercited by 1
  12. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
3 more
  1. 2026MultiHashFormer: Hash-based Generative Language Modelscited by 1
  2. 2026A Family of LLMs Liberated from Static Vocabulariescited by 0
  3. 2026Easy to anticipate, hard to compute: boundary dependence finds the computed outputs that entropy patching missescited by 0