# Melody Infilling with User-Provided Structural Context

arXiv: https://arxiv.org/abs/2210.02829 · HTML: https://mlsanity.com/paper/2210.02829/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation](https://mlsanity.com/paper/1703.10847/) (2017)
- [PopMAG: Pop Music Accompaniment Generation](https://mlsanity.com/paper/2008.07703/) (2020)
- [LakhNES: Improving multi-instrumental music generation with cross-domain pre-training](https://mlsanity.com/paper/1907.04868/) (2019)
- [The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures](https://mlsanity.com/paper/2008.01307/) (2020)
- [Counterpoint by Convolution](https://mlsanity.com/paper/1903.07227/) (2019)
- [Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer](https://mlsanity.com/paper/2111.04093/) (2021)
- [Relative Positional Encoding for Transformers with Linear Complexity](https://mlsanity.com/paper/2105.08399/) (2021)
- [Music transcription modelling and composition using deep learning](https://mlsanity.com/paper/1604.08723/) (2016)
- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [Controllable deep melody generation via hierarchical music structure representation](https://mlsanity.com/paper/2109.00663/) (2021)
- [Learning to Traverse Latent Spaces for Musical Score Inpainting](https://mlsanity.com/paper/1907.01164/) (2019)
- [Automatic Analysis and Influence of Hierarchical Structure on Melody, Rhythm and Harmony in Popular Music](https://mlsanity.com/paper/2010.07518/) (2020)
- [Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encoding](https://mlsanity.com/paper/2108.05064/) (2021)
- [NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting](https://mlsanity.com/paper/1907.10380/) (2019)
- [Algorithmic Composition of Melodies with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1606.07251/) (2016)
- [Music Score Expansion with Variable-Length Infilling](https://mlsanity.com/paper/2111.06046/) (2021)

## Built on by (most cited first, 4 of 4)

- [Anticipatory Music Transformer](https://mlsanity.com/paper/2306.08620/) (2023)
- [MelodyGLM: Multi-task Pre-training for Symbolic Melody Generation](https://mlsanity.com/paper/2309.10738/) (2023)
- [Segment-Factorized Full-Song Generation on Symbolic Piano Music](https://mlsanity.com/paper/2510.05881/) (2025)
- [Pictures Of MIDI: Controlled Music Generation via Graphical Prompts for Image-Based Diffusion Inpainting](https://mlsanity.com/paper/2407.01499/) (2024)

## Abstract (arXiv)

This paper proposes a novel Transformer-based model for music score infilling, to generate a music passage that fills in the gap between given past and future contexts. While existing infilling approaches can generate a passage that connects smoothly locally with the given contexts, they do not take into account the musical form or structure of the music and may therefore generate overly smooth results. To address this issue, we propose a structure-aware conditioning approach that employs a novel attention-selecting module to supply user-provided structure-related information to the Transformer for infilling. With both objective and subjective evaluations, we show that the proposed model can harness the structural information effectively and generate melodies in the style of pop of higher quality than the two existing structure-agnostic infilling models.