mlsanityData to 7 Oct 2026

Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models

arXiv 2403.11706 ↗First posted 18 Mar 2024cs.SD cs.LG eess.AScited by 13

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Multi-Source Diffusion Models (MSDM) allow for compositional musical generation tasks: generating a set of coherent sources, creating accompaniments, and performing source separation. Despite their versatility, they require estimating the joint distribution over the sources, necessitating pre-separated musical data, which is rarely available, and fixing the number and type of sources at training time. This paper generalizes MSDM to arbitrary time-domain diffusion models conditioned on text embeddings. These models do not require separated data as they are trained on mixtures, can parameterize an arbitrary number of sources, and allow for rich semantic control. We propose an inference procedure enabling the coherent generation of sources and accompaniments. Additionally, we adapt the Dirac separator of MSDM to perform source separation. We experiment with diffusion models trained on Slakh2100 and MTG-Jamendo, showcasing competitive generation and separation results in a relaxed data setting.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 24 of its 27 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

24 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models28 Aug
  2. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  3. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models3 Apr
  4. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separation4 Feb
  5. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusion27 Jan

Other references · 19

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  8. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  9. 2023Simple and Controllable Music Generationcited by 480
  10. 2023MusicLM: Generating Music From Textcited by 442
  11. 2018SDR - half-baked or well done?cited by 420
  12. 2023Stay on topic with Classifier-Free Guidancecited by 90
7 more
  1. 2019Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantitycited by 86
  2. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  3. 2023SingSong: Generating musical accompaniments from singingcited by 54
  4. 2020Source Separation with Deep Generative Priorscited by 35
  5. 2021Learning music audio representations via weak language supervisioncited by 13
  6. 2022Improving Source Separation by Explicitly Modeling Dependencies Between Sourcescited by 8
  7. 2023GASS: Generalizing Audio Source Separation with Large-scale Datacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 3; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  2. 2024Taming Data and Transformers for Audio Generationcited by 28
  3. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  4. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  5. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
  6. 2024Naturalistic Music Decoding from EEG Data via Latent Diffusion Modelscited by 8
  7. 2024Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problemscited by 7
  8. 2024Multi-Source Music Generation with Latent Diffusioncited by 4
  9. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  10. 2025ZeroSep: Separate Anything in Audio with Zero Trainingcited by 4
  11. 2025Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidancecited by 2
  12. 2024The Interpretation Gap in Text-to-Music Generation Modelscited by 1
1 more
  1. 2026Soundwich: Video Generation with Layered and Controllable Audiocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models
  2. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models
  3. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
  4. 2024Improving Music Source Separation with Diffusion and Consistency Refinement
  5. 2023ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models
  6. 2025User-guided Generative Source Separation
  7. 2021Symbolic Music Generation with Diffusion Models
  8. 2025Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning