# Multi-scale Transformer Language Models

arXiv: https://arxiv.org/abs/2005.00581 · HTML: https://mlsanity.com/paper/2005.00581/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 12 of 12)

- [Hierarchical Transformers Are More Efficient Language Models](https://mlsanity.com/paper/2110.13711/) (2021)
- [Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting](https://mlsanity.com/paper/2206.04038/) (2022)
- [SpaceByte: Towards Deleting Tokenization from Large Language Modeling](https://mlsanity.com/paper/2404.14408/) (2024)
- [Block-State Transformers](https://mlsanity.com/paper/2306.09539/) (2023)
- [MSHyper: Multi-Scale Hypergraph Transformer for Long-Range Time Series Forecasting](https://mlsanity.com/paper/2401.09261/) (2024)
- [Multi-Token Attention](https://mlsanity.com/paper/2504.00927/) (2025)
- [Characterizing Verbatim Short-Term Memory in Neural Language Models](https://mlsanity.com/paper/2210.13569/) (2022)
- [Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis](https://mlsanity.com/paper/2212.09162/) (2022)
- [DisMS-TS: Eliminating Redundant Multi-Scale Features for Time Series Classification](https://mlsanity.com/paper/2507.04600/) (2025)
- [Sequence Shortening for Context-Aware Machine Translation](https://mlsanity.com/paper/2402.01416/) (2024)
- [TimeMar: Multi-Scale Autoregressive Modeling for Unconditional Time Series Generation](https://mlsanity.com/paper/2601.11184/) (2026)
- [Hierarchical Resolution Transformers: A Wavelet-Inspired Architecture for Multi-Scale Language Understanding](https://mlsanity.com/paper/2509.20581/) (2025)

## Abstract (arXiv)

We investigate multi-scale transformer language models that learn representations of text at multiple scales, and present three different architectures that have an inductive bias to handle the hierarchical nature of language. Experiments on large-scale language modeling benchmarks empirically demonstrate favorable likelihood vs memory footprint trade-offs, e.g. we show that it is possible to train a hierarchical variant with 30 layers that has 23% smaller memory footprint and better perplexity, compared to a vanilla transformer with less than half the number of layers, on the Toronto BookCorpus. We analyze the advantages of learned representations at multiple scales in terms of memory footprint, compute time, and perplexity, which are particularly appealing given the quadratic scaling of transformers' run time and memory usage with respect to sequence length.