# MIDI-Draw: Sketching to Control Melody Generation

arXiv: https://arxiv.org/abs/2305.11605 · HTML: https://mlsanity.com/paper/2305.11605/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 8 of 8)

- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm](https://mlsanity.com/paper/2008.01291/) (2020)
- [SurpriseNet: Melody Harmonization Conditioning on User-controlled Surprise Contours](https://mlsanity.com/paper/2108.00378/) (2021)
- [Exploring Conditioning for Generative Music Systems with Human-Interpretable Controls](https://mlsanity.com/paper/1907.04352/) (2019)
- [A variational autoencoder for music generation controlled by tonal tension](https://mlsanity.com/paper/2010.06230/) (2020)
- [dMelodies: A Music Dataset for Disentanglement Learning](https://mlsanity.com/paper/2007.15067/) (2020)

## Abstract (arXiv)

We describe a proof-of-principle implementation of a system for drawing melodies that abstracts away from a note-level input representation via melodic contours. The aim is to allow users to express their musical intentions without requiring prior knowledge of how notes fit together melodiously. Current approaches to controllable melody generation often require users to choose parameters that are static across a whole sequence, via buttons or sliders. In contrast, our method allows users to quickly specify how parameters should change over time by drawing a contour.