# Piano Genie

arXiv: https://arxiv.org/abs/1810.05246 · HTML: https://mlsanity.com/paper/1810.05246/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2018.

## Built on by (most cited first, 11 of 11)

- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [Continuous diffusion for categorical data](https://mlsanity.com/paper/2211.15089/) (2022)
- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [Variable-rate discrete representation learning](https://mlsanity.com/paper/2103.06089/) (2021)
- [The Piano Inpainting Application](https://mlsanity.com/paper/2107.05944/) (2021)
- [Content-based Controls For Music Large Language Modeling](https://mlsanity.com/paper/2310.17162/) (2023)
- [Live Music Models](https://mlsanity.com/paper/2508.04651/) (2025)
- [Is Disentanglement enough? On Latent Representations for Controllable Music Generation](https://mlsanity.com/paper/2108.01450/) (2021)
- [Musical Speech: A Transformer-based Composition Tool](https://mlsanity.com/paper/2108.01043/) (2021)
- [Prevailing Research Areas for Music AI in the Era of Foundation Models](https://mlsanity.com/paper/2409.09378/) (2024)
- [Profy: Interpretable Visualization of Expertise-Dependent Motor Skills Toward Supporting Piano Practice](https://mlsanity.com/paper/2606.10627/) (2026)

## Abstract (arXiv)

We present Piano Genie, an intelligent controller which allows non-musicians to improvise on the piano. With Piano Genie, a user performs on a simple interface with eight buttons, and their performance is decoded into the space of plausible piano music in real time. To learn a suitable mapping procedure for this problem, we train recurrent neural network autoencoders with discrete bottlenecks: an encoder learns an appropriate sequence of buttons corresponding to a piano piece, and a decoder learns to map this sequence back to the original piece. During performance, we substitute a user's input for the encoder output, and play the decoder's prediction each time the user presses a button. To improve the intuitiveness of Piano Genie's performance behavior, we impose musically meaningful constraints over the encoder's outputs.