mlsanityData to 7 Oct 2026

Symbolic Music Generation with Diffusion Models

arXiv 2103.16091 ↗First posted 30 Mar 2021cs.SD cs.LG eess.AScited by 152

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Score-based generative models and diffusion probabilistic models have been successful at generating high-quality samples in continuous domains such as images and audio. However, due to their Langevin-inspired sampling mechanisms, their application to discrete and sequential data has been limited. In this work, we present a technique for training diffusion models on sequential data by parameterizing the discrete domain in the continuous latent space of a pre-trained variational autoencoder. Our method is non-autoregressive and learns to generate sequences of latent embeddings through the reverse process and offers parallel generation with a constant number of iterative refinement steps. We apply this technique to modeling symbolic music and show strong unconditional generation and post-hoc conditional infilling results compared to autoregressive language models operating over the same continuous embeddings.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 34 of its 60 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

34 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2016Layer Normalizationcited by 5,579
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2013Auto-Encoding Variational Bayescited by 3,980
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  12. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
21 more
  1. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  2. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  3. 2015Generating Sentences from a Continuous Spacecited by 607
  4. 2020Jukebox: A Generative Model for Musiccited by 535
  5. 2019On the "steerability" of generative adversarial networkscited by 247
  6. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  7. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 215
  8. 2018Music Transformercited by 207
  9. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  10. 2018A Hierarchical Latent Vector Model for Learning Long-Term Structure in Musiccited by 131
  11. 2019GANSynth: Adversarial Neural Audio Synthesiscited by 114
  12. 2020VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Modelscited by 103
  13. 2019Counterpoint by Convolutioncited by 42
  14. 2017Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Modelscited by 39
  15. 2019Encoding Musical Style with Transformer Autoencoderscited by 35
  16. 2020PIANOTREE VAE: Structured Representation Learning for Polyphonic Musiccited by 25
  17. 2019Implicit Generation and Generalization in Energy-Based Modelscited by 21
  18. 2020Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositionscited by 16
  19. 2018Piano Geniecited by 11
  20. 2018Learning a Latent Space of Multitrack Measurescited by 8
  21. 2016Imposing higher-level Structure in Polyphonic Music Generation using Convolutional Restricted Boltzmann Machines and Constraintscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 30; 2023: 46; 2024: 32; 2025: 19; 2026: 10.

152 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  4. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  5. 2022On Distillation of Guided Diffusion Modelscited by 721
  6. 2021Score-based Generative Modeling in Latent Spacecited by 634
  7. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  8. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  9. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  10. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  11. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  12. 2022Flexible Diffusion Modeling of Long Videoscited by 294
38 more
  1. 2022Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equationscited by 273
  2. 2022Post-training Quantization on Diffusion Modelscited by 262
  3. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 230
  4. 2021Gotta Go Fast When Generating Data with Score-Based Modelscited by 218
  5. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  6. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
  7. 2022Generative Modelling With Inverse Heat Dissipationcited by 164
  8. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillationcited by 115
  9. 2022Self-conditioned Embedding Diffusion for Text Generationcited by 114
  10. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  11. 2021D2C: Diffusion-Denoising Models for Few-shot Conditional Generationcited by 96
  12. 2023Input Perturbation Reduces Exposure Bias in Diffusion Modelscited by 88
  13. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  14. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  15. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separationcited by 63
  16. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  17. 2024Unveil Conditional Diffusion Models with Classifier-free Guidance: A Sharp Statistical Theorycited by 58
  18. 2021Diffusion-Based Representation Learningcited by 57
  19. 2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
  20. 2024WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Modelcited by 55
  21. 2022Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Modelscited by 50
  22. 2023Instruct 3D-to-3D: Text Instruction Guided 3D-to-3D conversioncited by 47
  23. 2022Probability flow solution of the Fokker-Planck equationcited by 46
  24. 2022Discrete Contrastive Diffusion for Cross-Modal Music and Image Generationcited by 44
  25. 2022DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANscited by 43
  26. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  27. 2024Accelerating Diffusion Models with Parallel Sampling: Inference at Sub-Linear Time Complexitycited by 42
  28. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 39
  29. 2023Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer modelcited by 37
  30. 2023Anticipatory Music Transformercited by 35
  31. 2023Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controlscited by 35
  32. 2024Foundation Models for Music: A Surveycited by 34
  33. 2023Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusionscited by 27
  34. 2024Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Trackingcited by 27
  35. 2023PLay: Parametrically Conditioned Layout Generation using Latent Diffusioncited by 24
  36. 2024Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Modelscited by 24
  37. 2023Regular Time-series Generation using SGMcited by 23
  38. 2023Boundary Guided Learning-Free Semantic Control with Diffusion Modelscited by 23

50 most cited of 152 All 152 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Amadeus: Autoregressive Model with Bidirectional Attribute Modelling for Symbolic Music
  2. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models
  3. 2024Long-form music generation with latent diffusion
  4. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
  5. 2024Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
  6. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes
  7. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models
  8. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models