# Towards Automatic Instrumentation by Learning to Separate Parts in Symbolic Multitrack Music

arXiv: https://arxiv.org/abs/2107.05916 · HTML: https://mlsanity.com/paper/2107.05916/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 43 references that are in the map.
- First posted on arXiv 13 Jul 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment](https://mlsanity.com/paper/1709.06298/) (2017)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [Learning Features of Music from Scratch](https://mlsanity.com/paper/1611.09827/) (2016)
- [LakhNES: Improving multi-instrumental music generation with cross-domain pre-training](https://mlsanity.com/paper/1907.04868/) (2019)
- [Deep Learning Techniques for Music Generation -- A Survey](https://mlsanity.com/paper/1709.01620/) (2017)
- [MMM : Exploring Conditional Multi-Track Music Generation with the Transformer](https://mlsanity.com/paper/2008.06048/) (2020)
- [MusPy: A Toolkit for Symbolic Music Generation](https://mlsanity.com/paper/2008.01951/) (2020)
- [Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions](https://mlsanity.com/paper/2002.00212/) (2020)
- [The NES Music Database: A multi-instrumental dataset with expressive performance attributes](https://mlsanity.com/paper/1806.04278/) (2018)
- [Improving Automatic Jazz Melody Generation by Transfer Learning Techniques](https://mlsanity.com/paper/1908.09484/) (2019)
- [Live Orchestral Piano, a system for real-time orchestral music generation](https://mlsanity.com/paper/1609.01203/) (2016)
- [Statistical Piano Reduction Controlling Performance Difficulty](https://mlsanity.com/paper/1808.05006/) (2018)
- [From Note-Level to Chord-Level Neural Network Models for Voice Separation in Symbolic Music](https://mlsanity.com/paper/2011.03028/) (2020)

## Built on by (most cited first, 5 of 5)

- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization](https://mlsanity.com/paper/2408.15176/) (2024)
- [Instrument Separation of Symbolic Music by Explicitly Guided Diffusion Model](https://mlsanity.com/paper/2209.02696/) (2022)
- [UOT-IR: Structured Routing of High-Polyphony Symbolic Music into Fixed-Budget Representations](https://mlsanity.com/paper/2608.00576/) (2026)

## Abstract (arXiv)

Modern keyboards allow a musician to play multiple instruments at the same time by assigning zones -- fixed pitch ranges of the keyboard -- to different instruments. In this paper, we aim to further extend this idea and examine the feasibility of automatic instrumentation -- dynamically assigning instruments to notes in solo music during performance. In addition to the online, real-time-capable setting for performative use cases, automatic instrumentation can also find applications in assistive composing tools in an offline setting. Due to the lack of paired data of original solo music and their full arrangements, we approach automatic instrumentation by learning to separate parts (e.g., voices, instruments and tracks) from their mixture in symbolic multitrack music, assuming that the mixture is to be played on a keyboard. We frame the task of part separation as a sequential multi-class classification problem and adopt machine learning to map sequences of notes into sequences of part labels. To examine the effectiveness of our proposed models, we conduct a comprehensive empirical evaluation over four diverse datasets of different genres and ensembles -- Bach chorales, string quartets, game music and pop music. Our experiments show that the proposed models outperform various baselines. We also demonstrate the potential for our proposed models to produce alternative convincing instrumentations for an existing arrangement by separating its mixture into parts. All source code and audio samples can be found at https://salu133445.github.io/arranger/ .