# Generating Separated Singing Vocals Using a Diffusion Model Conditioned on Music Mixtures

arXiv: https://arxiv.org/abs/2511.21342 · HTML: https://mlsanity.com/paper/2511.21342/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Hybrid Transformers for Music Source Separation](https://mlsanity.com/paper/2211.08553/) (2022)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [Mousai: Text-to-Music Generation with Long-Context Latent Diffusion](https://mlsanity.com/paper/2301.11757/) (2023)
- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [The 2018 Signal Separation Evaluation Campaign](https://mlsanity.com/paper/1804.06267/) (2018)
- [Music Source Separation with Band-Split RoPE Transformer](https://mlsanity.com/paper/2309.02612/) (2023)
- [D3Net: Densely connected multidilated DenseNet for music source separation](https://mlsanity.com/paper/2010.01733/) (2020)
- [The Sound Demixing Challenge 2023 unicodex2013 Music Demixing Track](https://mlsanity.com/paper/2308.06979/) (2023)
- [Unsupervised Audio Source Separation using Generative Priors](https://mlsanity.com/paper/2005.13769/) (2020)
- [Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models](https://mlsanity.com/paper/2409.12346/) (2024)
- [Latent Autoregressive Source Separation](https://mlsanity.com/paper/2301.08562/) (2023)
- [Music Source Separation with Generative Flow](https://mlsanity.com/paper/2204.09079/) (2022)
- [Music Separation Enhancement with Generative Modeling](https://mlsanity.com/paper/2208.12387/) (2022)
- [30+ Years of Source Separation Research: Achievements and Future Challenges](https://mlsanity.com/paper/2501.11837/) (2025)
- [Content Based Singing Voice Extraction From a Musical Mixture](https://mlsanity.com/paper/2002.04933/) (2020)
- [Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model](https://mlsanity.com/paper/2511.20470/) (2025)
- [Learned complex masks for multi-instrument source separation](https://mlsanity.com/paper/2103.12864/) (2021)

## Built on by (most cited first, 1 of 1)

- [Diff-VS: Efficient Audio-Aware Diffusion U-Net for Vocals Separation](https://mlsanity.com/paper/2604.01120/) (2026)

## Abstract (arXiv)

Separating the individual elements in a musical mixture is an essential process for music analysis and practice. While this is generally addressed using neural networks optimized to mask or transform the time-frequency representation of a mixture to extract the target sources, the flexibility and generalization capabilities of generative diffusion models are giving rise to a novel class of solutions for this complicated task. In this work, we explore singing voice separation from real music recordings using a diffusion model which is trained to generate the solo vocals conditioned on the corresponding mixture. Our approach improves upon prior generative systems and achieves competitive objective scores against non-generative baselines when trained with supplementary data. The iterative nature of diffusion sampling enables the user to control the quality-efficiency trade-off, and also refine the output when needed. We present an ablation study of the sampling algorithm, highlighting the effects of the user-configurable parameters.