mlsanityData to 14 Oct 2026

MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction

arXiv 2505.23305 ↗First posted 29 May 2025cs.SD cs.LG eess.AScited by 4

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We present MGE-LDM, a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation. Unlike prior approaches constrained to fixed instrument classes, MGE-LDM learns a joint distribution over full mixtures, submixtures, and individual stems within a single compact latent diffusion model. At inference, MGE-LDM enables (1) complete mixture generation, (2) partial generation (i.e., source imputation), and (3) text-conditioned extraction of arbitrary sources. By formulating both separation and imputation as conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources. Notably, MGE-LDM can be trained jointly across heterogeneous multi-track datasets (e.g., Slakh2100, MUSDB18, MoisesDB) without relying on predefined instrument categories. Audio samples are available at our project page: https://yoongi43.github.io/MGELDM_Samples/.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 102 of its 115 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

102 of 115 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Music2Latent2: Audio Compression with Summary Embeddings and Autoregressive Decoding29 Jan
  2. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformers30 Oct
  3. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models18 Sep
  4. 2024Multi-Source Music Generation with Latent Diffusion10 Sep
  5. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model4 Sep
  6. 2024Music2Latent: Consistency Autoencoders for Latent Audio Compression12 Aug
  7. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models12 Jun
  8. 2024Long-form music generation with latent diffusion16 Apr
  9. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models18 Mar
  10. 2024Bass Accompaniment Generation via Latent Diffusion2 Feb
  11. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  12. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies3 Aug
9 more
  1. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models3 Apr
  2. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  3. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separation4 Feb
  4. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  5. 2022Cold Diffusion for Speech Enhancement4 Nov
  6. 2022Diffusion-based Generative Speech Source Separation31 Oct
  7. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  8. 2022Conditional Diffusion Probabilistic Model for Speech Enhancement10 Feb
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 81

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2016Layer Normalizationcited by 5,669
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
38 more
  1. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  3. 2018Group Normalizationcited by 1,524
  4. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  5. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  6. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  8. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  9. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  10. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  11. 2022High Fidelity Neural Audio Compressioncited by 687
  12. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  13. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  14. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  15. 2020Jukebox: A Generative Model for Musiccited by 542
  16. 2021Image Segmentation Using Text and Image Promptscited by 512
  17. 2023Simple and Controllable Music Generationcited by 488
  18. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  19. 2023MusicLM: Generating Music From Textcited by 448
  20. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  21. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  22. 2022AudioGen: Textually Guided Audio Generationcited by 295
  23. 2018The Sound of Pixelscited by 259
  24. 2018Adversarial Audio Synthesiscited by 225
  25. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  26. 2016FMA: A Dataset For Music Analysiscited by 207
  27. 2024Stable Audio Opencited by 197
  28. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  29. 2019GANSynth: Adversarial Neural Audio Synthesiscited by 116
  30. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  31. 2019Music Source Separation in the Waveform Domaincited by 99
  32. 2023Adapting Frechet Audio Distance for Generative Music Evaluationcited by 92
  33. 2021Hybrid Spectrogram and Waveform Source Separationcited by 91
  34. 2019Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantitycited by 86
  35. 2023Music ControlNet: Multiple Time-varying Controls for Music Generationcited by 83
  36. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  37. 2022Music Source Separation with Band-split RNNcited by 70
  38. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSPcited by 2
  2. 2026Stemphonic: All-at-once Flexible Multi-stem Music Generationcited by 1
  3. 2026Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Modelscited by 0
  4. 2026Soundwich: Video Generation with Layered and Controllable Audiocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025User-guided Generative Source Separation
  2. 2024Improving Music Source Separation with Diffusion and Consistency Refinement
  3. 2026Diff-VS: Efficient Audio-Aware Diffusion U-Net for Vocals Separation
  4. 2024Unsupervised Composable Representations for Audio
  5. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes
  6. 2025Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model
  7. 2026S2Accompanist: A Semantic-Aware and Structure-Guided Diffusion Model for Music Accompaniment Generation
  8. 2023Efficient Neural Music Generation