# MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer

arXiv: https://arxiv.org/abs/1809.07600 · HTML: https://mlsanity.com/paper/1809.07600/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2018.

## Built on by (most cited first, 30 of 33)

- [Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion](https://mlsanity.com/paper/2402.14285/) (2024)
- [MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE](https://mlsanity.com/paper/2105.04090/) (2021)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Generating Lead Sheets with Affect: A Novel Conditional seq2seq Framework](https://mlsanity.com/paper/2104.13056/) (2021)
- [Long-Term Rhythmic Video Soundtracker](https://mlsanity.com/paper/2305.01319/) (2023)
- [Music Composition with Deep Learning: A Review](https://mlsanity.com/paper/2108.12290/) (2021)
- [A Review of Intelligent Music Generation Systems](https://mlsanity.com/paper/2211.09124/) (2022)
- [Music Style Transfer with Time-Varying Inversion of Diffusion Models](https://mlsanity.com/paper/2402.13763/) (2024)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [Is Disentanglement enough? On Latent Representations for Controllable Music Generation](https://mlsanity.com/paper/2108.01450/) (2021)
- [A Comprehensive Survey for Evaluation Methodologies of AI-Generated Music](https://mlsanity.com/paper/2308.13736/) (2023)
- [Review for Handling Missing Data with special missing mechanism](https://mlsanity.com/paper/2404.04905/) (2024)
- [The GigaMIDI Dataset with Features for Expressive Music Performance Detection](https://mlsanity.com/paper/2502.17726/) (2025)
- [Towards Robust Unsupervised Disentanglement of Sequential Data -- A Case Study Using Music Audio](https://mlsanity.com/paper/2205.05871/) (2022)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [An Order-Complexity Aesthetic Assessment Model for Aesthetic-aware Music Recommendation](https://mlsanity.com/paper/2402.08300/) (2024)
- [Learning long-term music representations via hierarchical contextual constraints](https://mlsanity.com/paper/2202.06180/) (2022)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training](https://mlsanity.com/paper/2302.00286/) (2023)
- [Graph-based Polyphonic Multitrack Music Generation](https://mlsanity.com/paper/2307.14928/) (2023)
- [A Survey of Music Generation in the Context of Interaction](https://mlsanity.com/paper/2402.15294/) (2024)
- [Applications and Advances of Artificial Intelligence in Music Generation:A Review](https://mlsanity.com/paper/2409.03715/) (2024)
- [Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes](https://mlsanity.com/paper/2110.05587/) (2021)
- [FIGARO: Generating Symbolic Music with Fine-Grained Artistic Control](https://mlsanity.com/paper/2201.10936/) (2022)
- [An Order-Complexity Model for Aesthetic Quality Assessment of Homophony Music Performance](https://mlsanity.com/paper/2304.11521/) (2023)
- [Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation](https://mlsanity.com/paper/2412.05296/) (2024)
- [Etude: Piano Cover Generation with a Three-Stage Approach -- Extract, strucTUralize, and DEcode](https://mlsanity.com/paper/2509.16522/) (2025)
- [InverseMV: Composing Piano Scores with a Convolutional Video-Music Transformer](https://mlsanity.com/paper/2112.15320/) (2021)
- [Can Machines Generate Personalized Music? A Hybrid Favorite-aware Method for User Preference Music Transfer](https://mlsanity.com/paper/2201.08526/) (2022)
- [Music Style Transfer With Diffusion Model](https://mlsanity.com/paper/2404.14771/) (2024)

## Abstract (arXiv)

We introduce MIDI-VAE, a neural network model based on Variational Autoencoders that is capable of handling polyphonic music with multiple instrument tracks, as well as modeling the dynamics of music by incorporating note durations and velocities. We show that MIDI-VAE can perform style transfer on symbolic music by automatically changing pitches, dynamics and instruments of a music piece from, e.g., a Classical to a Jazz style. We evaluate the efficacy of the style transfer by training separate style validation classifiers. Our model can also interpolate between short pieces of music, produce medleys and create mixtures of entire songs. The interpolations smoothly change pitches, dynamics and instrumentation to create a harmonic bridge between two music pieces. To the best of our knowledge, this work represents the first successful attempt at applying neural style transfer to complete musical compositions.