# MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Control

arXiv: https://arxiv.org/abs/2202.05528 · HTML: https://mlsanity.com/paper/2202.05528/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [This Time with Feeling: Learning Expressive Musical Performance](https://mlsanity.com/paper/1808.03715/) (2018)
- [PopMAG: Pop Music Accompaniment Generation](https://mlsanity.com/paper/2008.07703/) (2020)
- [A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions](https://mlsanity.com/paper/2011.06801/) (2020)
- [MMM : Exploring Conditional Multi-Track Music Generation with the Transformer](https://mlsanity.com/paper/2008.06048/) (2020)
- [Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling](https://mlsanity.com/paper/2007.15474/) (2020)
- [Counterpoint by Convolution](https://mlsanity.com/paper/1903.07227/) (2019)
- [Symbolic Music Genre Transfer with CycleGAN](https://mlsanity.com/paper/1809.07575/) (2018)
- [Learning to Traverse Latent Spaces for Musical Score Inpainting](https://mlsanity.com/paper/1907.01164/) (2019)
- [The Piano Inpainting Application](https://mlsanity.com/paper/2107.05944/) (2021)
- [MorpheuS: generating structured music with constrained patterns and tension](https://mlsanity.com/paper/1812.04832/) (2018)
- [Hierarchical Recurrent Neural Networks for Conditional Melody Generation with Long-term Structure](https://mlsanity.com/paper/2102.09794/) (2021)
- [NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting](https://mlsanity.com/paper/1907.10380/) (2019)
- [A variational autoencoder for music generation controlled by tonal tension](https://mlsanity.com/paper/2010.06230/) (2020)
- [Is Disentanglement enough? On Latent Representations for Controllable Music Generation](https://mlsanity.com/paper/2108.01450/) (2021)
- [Midi Miner -- A Python library for tonal tension and track classification](https://mlsanity.com/paper/1910.02049/) (2019)
- [A Contextual Latent Space Model: Subsequence Modulation in Melodic Sequence](https://mlsanity.com/paper/2111.11703/) (2021)

## Built on by (most cited first, 17 of 17)

- [Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls](https://mlsanity.com/paper/2307.10304/) (2023)
- [A Review of Intelligent Music Generation Systems](https://mlsanity.com/paper/2211.09124/) (2022)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [Composer's Assistant: An Interactive Transformer for Multi-Track MIDI Infilling](https://mlsanity.com/paper/2301.12525/) (2023)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [Composer's Assistant 2: Interactive Multi-Track MIDI Infilling with Fine-Grained User Control](https://mlsanity.com/paper/2407.14700/) (2024)
- [Mamba-Diffusion Model with Learnable Wavelet for Controllable Symbolic Music Generation](https://mlsanity.com/paper/2505.03314/) (2025)
- [Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributes](https://mlsanity.com/paper/2505.15559/) (2025)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training](https://mlsanity.com/paper/2302.00286/) (2023)
- [From Words to Music: A Study of Subword Tokenization Techniques in Symbolic Music Generation](https://mlsanity.com/paper/2304.08953/) (2023)
- [BandCondiNet: Parallel Transformers-based Conditional Popular Music Generation with Multi-View Features](https://mlsanity.com/paper/2407.10462/) (2024)
- [WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion](https://mlsanity.com/paper/2409.15321/) (2024)
- [Adaptable Symbolic Music Infilling with MIDI-RWKV](https://mlsanity.com/paper/2506.13001/) (2025)
- [BeatEdit: Symbolic Music Generation as Explicit Editing](https://mlsanity.com/paper/2607.11124/) (2026)
- [Notochord: a Flexible Probabilistic Model for Real-Time MIDI Performance](https://mlsanity.com/paper/2403.12000/) (2024)
- [Explicit Tonal Tension Conditioning via Dual-Level Beam Search for Symbolic Music Generation](https://mlsanity.com/paper/2511.19342/) (2025)

## Abstract (arXiv)

We present a novel music generation framework for music infilling, with a user friendly interface. Infilling refers to the task of generating musical sections given the surrounding multi-track music. The proposed transformer-based framework is extensible for new control tokens as the added music control tokens such as tonal tension per bar and track polyphony level in this work. We explore the effects of including several musically meaningful control tokens, and evaluate the results using objective metrics related to pitch and rhythm. Our results demonstrate that adding additional control tokens helps to generate music with stronger stylistic similarities to the original music. It also provides the user with more control to change properties like the music texture and tonal tension in each bar compared to previous research which only provided control for track density. We present the model in a Google Colab notebook to enable interactive generation.