# Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

arXiv: https://arxiv.org/abs/1810.12247 · HTML: https://mlsanity.com/paper/1810.12247/

## Key facts

- Cited by 217 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2018.

## Built on by (most cited first, 30 of 217)

- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [General-purpose, long-context autoregressive modeling with Perceiver AR](https://mlsanity.com/paper/2202.07765/) (2022)
- [SingSong: Generating musical accompaniments from singing](https://mlsanity.com/paper/2301.12662/) (2023)
- [Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion](https://mlsanity.com/paper/2402.14285/) (2024)
- [Sequence-to-Sequence Piano Transcription with Transformers](https://mlsanity.com/paper/2107.09142/) (2021)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling](https://mlsanity.com/paper/2406.04321/) (2024)
- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models](https://mlsanity.com/paper/2402.06178/) (2024)
- [Video Background Music Generation: Dataset, Method and Evaluation](https://mlsanity.com/paper/2211.11248/) (2022)
- [A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation](https://mlsanity.com/paper/2203.09893/) (2022)
- [miditok: A Python package for MIDI file tokenization](https://mlsanity.com/paper/2310.17202/) (2023)
- [Unaligned Supervision For Automatic Music Transcription in The Wild](https://mlsanity.com/paper/2204.13668/) (2022)
- [Musika! Fast Infinite Waveform Music Generation](https://mlsanity.com/paper/2208.08706/) (2022)
- [Melody transcription via generative pre-training](https://mlsanity.com/paper/2212.01884/) (2022)
- [Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Models](https://mlsanity.com/paper/2502.11420/) (2025)
- [Automatic Piano Transcription with Hierarchical Frequency-Time Transformer](https://mlsanity.com/paper/2307.04305/) (2023)
- [Blind Audio Bandwidth Extension: A Diffusion-Based Zero-Shot Approach](https://mlsanity.com/paper/2306.01433/) (2023)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [DadaGP: A Dataset of Tokenized GuitarPro Songs for Sequence Models](https://mlsanity.com/paper/2107.14653/) (2021)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)

## Abstract (arXiv)

Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.