# DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability

arXiv: https://arxiv.org/abs/2210.05148 · HTML: https://mlsanity.com/paper/2210.05148/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 26 of 26)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment](https://mlsanity.com/paper/1709.06298/) (2017)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times](https://mlsanity.com/paper/2010.01815/) (2020)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [Unaligned Supervision For Automatic Music Transcription in The Wild](https://mlsanity.com/paper/2204.13668/) (2022)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [The Effect of Spectrogram Reconstruction on Automatic Music Transcription: An Alternative Approach to Improve Transcription Accuracy](https://mlsanity.com/paper/2010.09969/) (2020)
- [Revisiting the Onsets and Frames Model with Additive Attention](https://mlsanity.com/paper/2104.06607/) (2021)
- [Deep Unsupervised Drum Transcription](https://mlsanity.com/paper/1906.03697/) (2019)
- [HPPNet: Modeling the Harmonic Structure and Pitch Invariance in Piano Transcription](https://mlsanity.com/paper/2208.14339/) (2022)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [The impact of Audio input representations on neural network based music transcription](https://mlsanity.com/paper/2001.09989/) (2020)
- [nnAudio: An on-the-fly GPU Audio to Spectrogram Conversion Toolbox Using 1D Convolution Neural Networks](https://mlsanity.com/paper/1912.12055/) (2019)

## Built on by (most cited first, 13 of 13)

- [Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion](https://mlsanity.com/paper/2310.02279/) (2023)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model](https://mlsanity.com/paper/2311.00968/) (2023)
- [Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls](https://mlsanity.com/paper/2307.10304/) (2023)
- [DExter: Learning and Controlling Performance Expression with Diffusion Models](https://mlsanity.com/paper/2406.14850/) (2024)
- [PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance](https://mlsanity.com/paper/2406.09326/) (2024)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [D3RM: A Discrete Denoising Diffusion Refinement Model for Piano Transcription](https://mlsanity.com/paper/2501.05068/) (2025)
- [Annotation-free Automatic Music Transcription with Scalable Synthetic Data and Adversarial Domain Confusion](https://mlsanity.com/paper/2312.10402/) (2023)
- [MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakage](https://mlsanity.com/paper/2403.10024/) (2024)
- [Why Perturbing Symbolic Music is Necessary: Fitting the Distribution of Never-used Notes through a Joint Probabilistic Diffusion Model](https://mlsanity.com/paper/2408.01950/) (2024)
- [Noise-to-Notes: Diffusion-based Generation and Refinement for Automatic Drum Transcription](https://mlsanity.com/paper/2509.21739/) (2025)
- [Playability-Aware Audio-to-Tablature Guitar Transcription via Diffusion Models](https://mlsanity.com/paper/2608.30854/) (2026)

## Abstract (arXiv)

In this paper we propose a novel generative approach, DiffRoll, to tackle automatic music transcription (AMT). Instead of treating AMT as a discriminative task in which the model is trained to convert spectrograms into piano rolls, we think of it as a conditional generative task where we train our model to generate realistic looking piano rolls from pure Gaussian noise conditioned on spectrograms. This new AMT formulation enables DiffRoll to transcribe, generate and even inpaint music. Due to the classifier-free nature, DiffRoll is also able to be trained on unpaired datasets where only piano rolls are available. Our experiments show that DiffRoll outperforms its discriminative counterpart by 19 percentage points (ppt.) and our ablation studies also indicate that it outperforms similar existing methods by 4.8 ppt. Source code and demonstration are available https://sony.github.io/DiffRoll/.