# LakhNES: Improving multi-instrumental music generation with cross-domain pre-training

arXiv: https://arxiv.org/abs/1907.04868 · HTML: https://mlsanity.com/paper/1907.04868/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2019.

## Built on by (most cited first, 30 of 54)

- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation](https://mlsanity.com/paper/2210.10349/) (2022)
- [MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE](https://mlsanity.com/paper/2105.04090/) (2021)
- [Relative Positional Encoding for Transformers with Linear Complexity](https://mlsanity.com/paper/2105.08399/) (2021)
- [Anticipatory Music Transformer](https://mlsanity.com/paper/2306.08620/) (2023)
- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls](https://mlsanity.com/paper/2307.10304/) (2023)
- [NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigms](https://mlsanity.com/paper/2502.18008/) (2025)
- [Multitrack Music Transformer](https://mlsanity.com/paper/2207.06983/) (2022)
- [miditok: A Python package for MIDI file tokenization](https://mlsanity.com/paper/2310.17202/) (2023)
- [The Piano Inpainting Application](https://mlsanity.com/paper/2107.05944/) (2021)
- [Byte Pair Encoding for Symbolic Music](https://mlsanity.com/paper/2301.11975/) (2023)
- [Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encoding](https://mlsanity.com/paper/2108.05064/) (2021)
- [Symbolic music generation conditioned on continuous-valued emotions](https://mlsanity.com/paper/2203.16165/) (2022)
- [SongDriver: Real-time Music Accompaniment Generation without Logical Latency nor Exposure Bias](https://mlsanity.com/paper/2209.06054/) (2022)
- [Music Composition with Deep Learning: A Review](https://mlsanity.com/paper/2108.12290/) (2021)
- [JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VAE](https://mlsanity.com/paper/2210.06007/) (2022)
- [A Review of Intelligent Music Generation Systems](https://mlsanity.com/paper/2211.09124/) (2022)
- [MelodyGLM: Multi-task Pre-training for Symbolic Melody Generation](https://mlsanity.com/paper/2309.10738/) (2023)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [Survey on the Evaluation of Generative Models in Music](https://mlsanity.com/paper/2506.05104/) (2025)
- [Composer's Assistant: An Interactive Transformer for Multi-Track MIDI Infilling](https://mlsanity.com/paper/2301.12525/) (2023)
- [ComMU: Dataset for Combinatorial Music Generation](https://mlsanity.com/paper/2211.09385/) (2022)
- [YM2413-MDB: A Multi-Instrumental FM Video Game Music Dataset with Emotion Annotations](https://mlsanity.com/paper/2211.07131/) (2022)
- [A Comprehensive Survey for Evaluation Methodologies of AI-Generated Music](https://mlsanity.com/paper/2308.13736/) (2023)
- [Motifs, Phrases, and Beyond: The Modelling of Structure in Symbolic Music Generation](https://mlsanity.com/paper/2403.07995/) (2024)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Impact of time and note duration tokenizations on deep learning symbolic music modeling](https://mlsanity.com/paper/2310.08497/) (2023)
- [Towards Automatic Instrumentation by Learning to Separate Parts in Symbolic Multitrack Music](https://mlsanity.com/paper/2107.05916/) (2021)

## Abstract (arXiv)

We are interested in the task of generating multi-instrumental music scores. The Transformer architecture has recently shown great promise for the task of piano score generation; here we adapt it to the multi-instrumental setting. Transformers are complex, high-dimensional language models which are capable of capturing long-term structure in sequence data, but require large amounts of data to fit. Their success on piano score generation is partially explained by the large volumes of symbolic data readily available for that domain. We leverage the recently-introduced NES-MDB dataset of four-instrument scores from an early video game sound synthesis chip (the NES), which we find to be well-suited to training with the Transformer architecture. To further improve the performance of our model, we propose a pre-training technique to leverage the information in a large collection of heterogeneous music, namely the Lakh MIDI dataset. Despite differences between the two corpora, we find that this transfer learning procedure improves both quantitative and qualitative performance for our primary task.