# Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models

arXiv: https://arxiv.org/abs/2501.10322 · HTML: https://mlsanity.com/paper/2501.10322/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2025.

## Built on by (most cited first, 22 of 22)

- [From Bytes to Ideas: Language Modeling with Autoregressive U-Nets](https://mlsanity.com/paper/2506.14761/) (2025)
- [SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion](https://mlsanity.com/paper/2602.11698/) (2026)
- [Learning to Skip the Middle Layers of Transformers](https://mlsanity.com/paper/2506.21103/) (2025)
- [Continuous Autoregressive Language Models](https://mlsanity.com/paper/2510.27688/) (2025)
- [Bolmo: Byteifying the Next Generation of Language Models](https://mlsanity.com/paper/2512.15586/) (2025)
- [Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation](https://mlsanity.com/paper/2505.00022/) (2025)
- [Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models](https://mlsanity.com/paper/2601.13260/) (2026)
- [Do LLM hallucination detectors suffer from low-resource effect?](https://mlsanity.com/paper/2601.16766/) (2026)
- [Compute Optimal Tokenization](https://mlsanity.com/paper/2605.01188/) (2026)
- [Proxy Compression for Language Modeling](https://mlsanity.com/paper/2602.04289/) (2026)
- [Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness](https://mlsanity.com/paper/2603.10771/) (2026)
- [GQ-VAE: A gated quantized VAE for learning variable length tokens](https://mlsanity.com/paper/2512.21913/) (2025)
- [Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models](https://mlsanity.com/paper/2605.09630/) (2026)
- [When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring](https://mlsanity.com/paper/2510.04302/) (2025)
- [From Characters to Tokens: Dynamic Grouping with Hierarchical BPE](https://mlsanity.com/paper/2510.15517/) (2025)
- [LIME: Making LLM Data More Efficient with Linguistic Metadata Embeddings](https://mlsanity.com/paper/2512.07522/) (2025)
- [Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages](https://mlsanity.com/paper/2602.21377/) (2026)
- [Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model](https://mlsanity.com/paper/2605.30080/) (2026)
- [Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES](https://mlsanity.com/paper/2607.05691/) (2026)
- [Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content](https://mlsanity.com/paper/2607.16117/) (2026)
- [BCMT: Blockwise Causal Memory Transformer](https://mlsanity.com/paper/2608.13578/) (2026)
- [Dynamic Multi-Byte Prediction With Hierarchical Language Models](https://mlsanity.com/paper/2608.15454/) (2026)

## Abstract (arXiv)

Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tokenizers have become the de-facto standard, they present challenges such as large vocabularies, limited adaptability to new domains or languages, and sensitivity to spelling errors and variations. To overcome these limitations, we investigate a hierarchical architecture for autoregressive language modelling that combines character-level and word-level processing. It employs a lightweight character-level encoder to convert character sequences into word embeddings, which are then processed by a word-level backbone model and decoded back into characters via a compact character-level decoder. This method retains the sequence compression benefits of word-level tokenization without relying on a rigid, predefined vocabulary. We demonstrate, at scales up to 7 billion parameters, that hierarchical transformers match the downstream task performance of subword-tokenizer-based models while exhibiting significantly greater robustness to input perturbations. Additionally, during continued pretraining on an out-of-domain language, our model trains almost twice as fast, achieves superior performance on the target language, and retains more of its previously learned knowledge. Hierarchical transformers pave the way for NLP systems that are more robust, flexible, and generalizable across languages and domains.