# Composer's Assistant: An Interactive Transformer for Multi-Track MIDI Infilling

arXiv: https://arxiv.org/abs/2301.12525 · HTML: https://mlsanity.com/paper/2301.12525/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 29 Jan 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?](https://mlsanity.com/paper/2207.10551/) (2022)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [LakhNES: Improving multi-instrumental music generation with cross-domain pre-training](https://mlsanity.com/paper/1907.04868/) (2019)
- [The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures](https://mlsanity.com/paper/2008.01307/) (2020)
- [MMM : Exploring Conditional Multi-Track Music Generation with the Transformer](https://mlsanity.com/paper/2008.06048/) (2020)
- [Counterpoint by Convolution](https://mlsanity.com/paper/1903.07227/) (2019)
- [Investigating Efficiently Extending Transformers for Long Input Summarization](https://mlsanity.com/paper/2208.04347/) (2022)
- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm](https://mlsanity.com/paper/2008.01291/) (2020)
- [Learning to Traverse Latent Spaces for Musical Score Inpainting](https://mlsanity.com/paper/1907.01164/) (2019)
- [The Piano Inpainting Application](https://mlsanity.com/paper/2107.05944/) (2021)
- [MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Control](https://mlsanity.com/paper/2202.05528/) (2022)
- [Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encoding](https://mlsanity.com/paper/2108.05064/) (2021)
- [NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting](https://mlsanity.com/paper/1907.10380/) (2019)
- [The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling](https://mlsanity.com/paper/2209.14458/) (2022)

## Built on by (most cited first, 9 of 9)

- [Composer's Assistant 2: Interactive Multi-Track MIDI Infilling with Fine-Grained User Control](https://mlsanity.com/paper/2407.14700/) (2024)
- [Hookpad Aria: A Copilot for Songwriters](https://mlsanity.com/paper/2502.08122/) (2025)
- [Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization](https://mlsanity.com/paper/2408.15176/) (2024)
- [Adaptable Symbolic Music Infilling with MIDI-RWKV](https://mlsanity.com/paper/2506.13001/) (2025)
- [MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Models](https://mlsanity.com/paper/2511.03942/) (2025)
- [Stemphonic: All-at-once Flexible Multi-stem Music Generation](https://mlsanity.com/paper/2602.09891/) (2026)
- [TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structure](https://mlsanity.com/paper/2506.23094/) (2025)
- [Barwise Section Boundary Detection in Symbolic Music Using Convolutional Neural Networks](https://mlsanity.com/paper/2509.16566/) (2025)
- [From One-Shot Generation to Incremental Music Composition: Adapting a General-Purpose Instruction LLM for Persistent Symbolic Editing](https://mlsanity.com/paper/2609.34994/) (2026)

## Abstract (arXiv)

We introduce Composer's Assistant, a system for interactive human-computer composition in the REAPER digital audio workstation. We consider the task of multi-track MIDI infilling when arbitrary track-measures have been deleted from a contiguous slice of measures from a MIDI file, and we train a T5-like model to accomplish this task. Composer's Assistant consists of this model together with scripts that enable interaction with the model in REAPER. We conduct objective and subjective tests of our model. We release our complete system, consisting of source code, pretrained models, and REAPER scripts. Our models were trained only on permissively-licensed MIDI files.