mlsanityData to 14 Oct 2026

Multi-Source Diffusion Models for Simultaneous Music Generation and Separation

arXiv 2302.02257 ↗First posted 4 Feb 2023cs.SD cs.LG eess.AScited by 63

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In this work, we define a diffusion-based generative model capable of both music synthesis and source separation by learning the score of the joint probability density of sources sharing a context. Alongside the classic total inference tasks (i.e., generating a mixture, separating the sources), we also introduce and experiment on the partial generation task of source imputation, where we generate a subset of the sources given the others (e.g., play a piano track that goes well with the drums). Additionally, we introduce a novel inference method for the separation task based on Dirac likelihood functions. We train our model on Slakh2100, a standard dataset for musical source separation, provide qualitative results in the generation settings, and showcase competitive quantitative results in the source separation setting. Our method is the first example of a single model that can handle both generation and separation tasks, thus representing a step toward general audio models.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 65 of its 83 references that are in the map. First posted on arXiv 4 Feb 2023.

Builds on

65 of 83 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  2. 2023Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation25 Jan
  3. 2022Unsupervised vocal dereverberation with diffusion-based generative models8 Nov
  4. 2022Diffusion-based Generative Speech Source Separation31 Oct
  5. 2022Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement27 Oct
  6. 2022Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution27 Oct
  7. 2022DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability11 Oct
  8. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  9. 2022Multi-instrument Music Synthesis with Spectrogram Diffusion11 Jun
  10. 2021A Study on Speech Enhancement Based on Diffusion Probabilistic Model25 Jul
  11. 2021CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis14 Jun
  12. 2021NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling6 Apr
2 more
  1. 2021Symbolic Music Generation with Diffusion Models30 Mar
  2. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
38 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  5. 2023Extracting Training Data from Diffusion Modelscited by 732
  6. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  7. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  9. 2020Jukebox: A Generative Model for Musiccited by 542
  10. 2023MusicLM: Generating Music From Textcited by 448
  11. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  12. 2018SDR - half-baked or well done?cited by 423
  13. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  14. 2022AudioGen: Textually Guided Audio Generationcited by 295
  15. 2018Adversarial Audio Synthesiscited by 225
  16. 2019Music Source Separation in the Waveform Domaincited by 99
  17. 2021Hybrid Spectrogram and Waveform Source Separationcited by 91
  18. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 87
  19. 2019Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantitycited by 86
  20. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  21. 2023SingSong: Generating musical accompaniments from singingcited by 55
  22. 2019Universal Sound Separationcited by 41
  23. 2020Source Separation with Deep Generative Priorscited by 36
  24. 2022Full-band General Audio Synthesis with Score-based Diffusioncited by 32
  25. 2017Generative Adversarial Source Separationcited by 19
  26. 2018MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separationcited by 19
  27. 2018End-to-end music source separation: is it possible in the waveform domain?cited by 19
  28. 2021Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamicscited by 19
  29. 2021A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesiscited by 18
  30. 2020Unsupervised Audio Source Separation using Generative Priorscited by 17
  31. 2022On loss functions and evaluation metrics for music source separationcited by 17
  32. 2022Evaluating generative audio systems and their metricscited by 16
  33. 2020LaSAFT: Latent Source Attentive Frequency Transformation for Conditioned Source Separationcited by 15
  34. 2021Learning music audio representations via weak language supervisioncited by 13
  35. 2023Latent Autoregressive Source Separationcited by 12
  36. 2022Music Source Separation with Generative Flowcited by 10
  37. 2019Single-Channel Signal Separation and Deconvolution with Generative Adversarial Networkscited by 9
  38. 2022Improving Source Separation by Explicitly Modeling Dependencies Between Sourcescited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 25; 2025: 16; 2026: 15.

63 papers.

Sort:

By area:

  1. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  2. 2024Long-form music generation with latent diffusioncited by 89
  3. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  4. 2023StemGen: A music generation model that listenscited by 32
  5. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  6. 2023JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generationcited by 17
  7. 2024Bass Accompaniment Generation via Latent Diffusioncited by 16
  8. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  9. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  10. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Modelscited by 13
  11. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Modelscited by 13
  12. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformercited by 13
38 more
  1. 2024SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformercited by 11
  2. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
  3. 2024Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfercited by 9
  4. 2023Score-based Source Separation with Applications to Digital Communication Signalscited by 8
  5. 2024Naturalistic Music Decoding from EEG Data via Latent Diffusion Modelscited by 8
  6. 2025Source Separation by Flow Matchingcited by 8
  7. 2024Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problemscited by 7
  8. 2025ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Priorcited by 7
  9. 2023Zero-Shot Duet Singing Voices Separation with Diffusion Modelscited by 6
  10. 2024FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generationcited by 5
  11. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Modelcited by 5
  12. 2024FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignmentcited by 5
  13. 2024CollaFuse: Collaborative Diffusion Modelscited by 4
  14. 2024Unsupervised Composable Representations for Audiocited by 4
  15. 2024Multi-Source Music Generation with Latent Diffusioncited by 4
  16. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  17. 2025LiLAC: A Lightweight Latent ControlNet for Musical Audio Generationcited by 3
  18. 2025Bayesian Signal Separation via Plug-and-Play Diffusion-Within-Gibbs Samplingcited by 3
  19. 2025Unsupervised Single-Channel Audio Separation with Diffusion Source Priorscited by 3
  20. 2024Why Perturbing Symbolic Music is Necessary: Fitting the Distribution of Never-used Notes through a Joint Probabilistic Diffusion Modelcited by 2
  21. 2025Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidancecited by 2
  22. 2025User-guided Generative Source Separationcited by 2
  23. 2025Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Modelcited by 2
  24. 2026Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSPcited by 2
  25. 2023Distribution Preserving Source Separation With Time Frequency Predictive Modelscited by 1
  26. 2024Weighted-Sum of Gaussian Process Latent Variable Modelscited by 1
  27. 2024Audio Decoding by Inverse Problem Solvingcited by 1
  28. 2024Improving Music Source Separation with Diffusion and Consistency Refinementcited by 1
  29. 2025Training-Free Multi-Step Audio Source Separationcited by 1
  30. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementationcited by 1
  31. 2025Learning Linearity in Audio Consistency Autoencoders via Implicit Regularizationcited by 1
  32. 2025Generating Separated Singing Vocals Using a Diffusion Model Conditioned on Music Mixturescited by 1
  33. 2026Conditional Flow Matching for Visually-Guided Acoustic Highlightingcited by 1
  34. 2026Stemphonic: All-at-once Flexible Multi-stem Music Generationcited by 1
  35. 2026Bayesian Signal Component Decomposition via Diffusion-within-Gibbs Samplingcited by 1
  36. 2023Multiple Noises in Diffusion Model for Semi-Supervised Multi-Domain Translationcited by 0
  37. 2024Source Separation of Multi-source Raw Music using a Residual Quantized Variational Autoencodercited by 0
  38. 2025LoopGen: Training-Free Loopable Music Generationcited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling
  2. 2026Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators
  3. 2023High-Quality Visually-Guided Sound Separation from Diverse Categories
  4. 2023ArchiSound: Audio Generation with Diffusion
  5. 2023Controllable Music Production with Diffusion Models and Guidance Gradients
  6. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models
  7. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond
  8. 2023Generating symbolic music using diffusion models