# The Piano Inpainting Application

arXiv: https://arxiv.org/abs/2107.05944 · HTML: https://mlsanity.com/paper/2107.05944/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 13 Jul 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [Image inpainting: A review](https://mlsanity.com/paper/1909.06399/) (2019)
- [This Time with Feeling: Learning Expressive Musical Performance](https://mlsanity.com/paper/1808.03715/) (2018)
- [High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times](https://mlsanity.com/paper/2010.01815/) (2020)
- [LakhNES: Improving multi-instrumental music generation with cross-domain pre-training](https://mlsanity.com/paper/1907.04868/) (2019)
- [GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music](https://mlsanity.com/paper/2010.07061/) (2020)
- [Learning to Traverse Latent Spaces for Musical Score Inpainting](https://mlsanity.com/paper/1907.01164/) (2019)
- [GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures](https://mlsanity.com/paper/1707.04588/) (2017)
- [Piano Genie](https://mlsanity.com/paper/1810.05246/) (2018)
- [NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting](https://mlsanity.com/paper/1907.10380/) (2019)
- [Vector Quantized Contrastive Predictive Coding for Template-based Music Generation](https://mlsanity.com/paper/2004.10120/) (2020)
- [Modelling Symbolic Music: Beyond the Piano Roll](https://mlsanity.com/paper/1606.01368/) (2016)

## Built on by (most cited first, 23 of 23)

- [miditok: A Python package for MIDI file tokenization](https://mlsanity.com/paper/2310.17202/) (2023)
- [Byte Pair Encoding for Symbolic Music](https://mlsanity.com/paper/2301.11975/) (2023)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Control](https://mlsanity.com/paper/2202.05528/) (2022)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [Composer's Assistant: An Interactive Transformer for Multi-Track MIDI Infilling](https://mlsanity.com/paper/2301.12525/) (2023)
- [SampleMatch: Drum Sample Retrieval by Musical Context](https://mlsanity.com/paper/2208.01141/) (2022)
- [Controlling Surprisal in Music Generation via Information Content Curve Matching](https://mlsanity.com/paper/2408.06022/) (2024)
- [Impact of time and note duration tokenizations on deep learning symbolic music modeling](https://mlsanity.com/paper/2310.08497/) (2023)
- [Composer's Assistant 2: Interactive Multi-Track MIDI Infilling with Fine-Grained User Control](https://mlsanity.com/paper/2407.14700/) (2024)
- [PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations](https://mlsanity.com/paper/2410.02060/) (2024)
- [Hookpad Aria: A Copilot for Songwriters](https://mlsanity.com/paper/2502.08122/) (2025)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [ImprovNet -- Generating Controllable Musical Improvisations with Iterative Corruption Refinement](https://mlsanity.com/paper/2502.04522/) (2025)
- [Xi-Net: Transformer Based Seismic Waveform Reconstructor](https://mlsanity.com/paper/2406.16932/) (2024)
- [Music Proofreading with RefinPaint: Where and How to Modify Compositions given Context](https://mlsanity.com/paper/2407.09099/) (2024)
- [TheGlueNote: Learned Representations for Robust and Flexible Note Alignment](https://mlsanity.com/paper/2408.04309/) (2024)
- [MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding](https://mlsanity.com/paper/2510.16273/) (2025)
- [Hidden Biases in Conditioning Autoregressive Models](https://mlsanity.com/paper/2604.07855/) (2026)
- [BeatEdit: Symbolic Music Generation as Explicit Editing](https://mlsanity.com/paper/2607.11124/) (2026)
- [Exploring Softly Masked Language Modelling for Controllable Symbolic Music Generation](https://mlsanity.com/paper/2305.03530/) (2023)
- [Analyzing Byte-Pair Encoding on Monophonic and Polyphonic Symbolic Music: A Focus on Musical Phrase Segmentation](https://mlsanity.com/paper/2410.01448/) (2024)
- [MidiTok Visualizer: a tool for visualization and analysis of tokenized MIDI symbolic music](https://mlsanity.com/paper/2410.20518/) (2024)

## Abstract (arXiv)

Autoregressive models are now capable of generating high-quality minute-long expressive MIDI piano performances. Even though this progress suggests new tools to assist music composition, we observe that generative algorithms are still not widely used by artists due to the limited control they offer, prohibitive inference times or the lack of integration within musicians' workflows. In this work, we present the Piano Inpainting Application (PIA), a generative model focused on inpainting piano performances, as we believe that this elementary operation (restoring missing parts of a piano performance) encourages human-machine interaction and opens up new ways to approach music composition. Our approach relies on an encoder-decoder Linear Transformer architecture trained on a novel representation for MIDI piano performances termed Structured MIDI Encoding. By uncovering an interesting synergy between Linear Transformers and our inpainting task, we are able to efficiently inpaint contiguous regions of a piano performance, which makes our model suitable for interactive and responsive A.I.-assisted composition. Finally, we introduce our freely-available Ableton Live PIA plugin, which allows musicians to smoothly generate or modify any MIDI clip using PIA within a widely-used professional Digital Audio Workstation.