# Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training

arXiv: https://arxiv.org/abs/2604.21265 · HTML: https://mlsanity.com/paper/2604.21265/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Builds on (most cited first, 6 of 6)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [Training Language Models via Neural Cellular Automata](https://mlsanity.com/paper/2603.10055/) (2026)

## Built on by (most cited first, 1 of 1)

- [Procedural Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2609.17831/) (2026)

## Abstract (arXiv)

We show that pre-training a Transformer on music before language significantly accelerates language acquisition. Using piano performances (MAESTRO dataset), a developmental pipeline -- music → poetry → prose -- yields a 17.5% perplexity improvement over random initialization (p < 0.001, 5 seeds), with music and poetry improving orthogonal model components (internal computation and embeddings, respectively). Convergence tests confirm that this is not a transient head start: at d!=!64, multi-seed validation (5 seeds) shows a persistent 5.5% gap at plateau (p = 0.017), with the pipeline converging faster and to a lower loss in every run. Real music matches the transfer ceiling of synthetic patterns with one-third the data, and scaling experiments reveal that optimal pre-training data volume shifts with model capacity (-3% → +3% → +6% advantage of larger datasets from d!=!16 to d!=!64). Across the scales we study (d!∈!16,32,64, up to ~400K parameters), these results suggest a capacity-dependent data curation principle and indicate that structured human creative outputs can provide an efficient pre-training substrate for small language models; stronger conclusions at modern pre-training scale will require substantially larger experiments.