# Zero-Shot Duet Singing Voices Separation with Diffusion Models

arXiv: https://arxiv.org/abs/2311.07345 · HTML: https://mlsanity.com/paper/2311.07345/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Mousai: Text-to-Music Generation with Long-Context Latent Diffusion](https://mlsanity.com/paper/2301.11757/) (2023)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus](https://mlsanity.com/paper/2112.10358/) (2021)
- [GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration](https://mlsanity.com/paper/2301.12686/) (2023)
- [Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis](https://mlsanity.com/paper/2201.07429/) (2022)
- [Asteroid: the PyTorch-based audio source separation toolkit for researchers](https://mlsanity.com/paper/2005.04132/) (2020)
- [CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis](https://mlsanity.com/paper/2106.07431/) (2021)
- [Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution](https://mlsanity.com/paper/2210.15793/) (2022)
- [Unsupervised vocal dereverberation with diffusion-based generative models](https://mlsanity.com/paper/2211.04124/) (2022)
- [JVS-MuSiC: Japanese multispeaker singing-voice corpus](https://mlsanity.com/paper/2001.07044/) (2020)
- [VRDMG: Vocal Restoration via Diffusion Posterior Sampling with Multiple Guidance](https://mlsanity.com/paper/2309.06934/) (2023)
- [PJS: phoneme-balanced Japanese singing voice corpus](https://mlsanity.com/paper/2006.02959/) (2020)
- [Transfer Learning with Jukebox for Music Source Separation](https://mlsanity.com/paper/2111.14200/) (2021)
- [Unsupervised Music Source Separation Using Differentiable Parametric Source Models](https://mlsanity.com/paper/2201.09592/) (2022)
- [MedleyVox: An Evaluation Dataset for Multiple Singing Voices Separation](https://mlsanity.com/paper/2211.07302/) (2022)
- [Diffusion-based Signal Refiner for Speech Enhancement and Separation](https://mlsanity.com/paper/2305.05857/) (2023)
- [Multi-layered Cepstrum for Instantaneous Frequency Estimation](https://mlsanity.com/paper/1902.00539/) (2019)
- [Unsupervised Source Separation By Steering Pretrained Music Models](https://mlsanity.com/paper/2110.13071/) (2021)

## Built on by (most cited first, 6 of 6)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models](https://mlsanity.com/paper/2409.12346/) (2024)
- [ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior](https://mlsanity.com/paper/2505.05657/) (2025)
- [Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model](https://mlsanity.com/paper/2511.20470/) (2025)
- [Improving Music Source Separation with Diffusion and Consistency Refinement](https://mlsanity.com/paper/2412.06965/) (2024)
- [Unsupervised Multi-channel Speech Dereverberation via Diffusion](https://mlsanity.com/paper/2508.02071/) (2025)

## Abstract (arXiv)

In recent studies, diffusion models have shown promise as priors for solving audio inverse problems. These models allow us to sample from the posterior distribution of a target signal given an observed signal by manipulating the diffusion process. However, when separating audio sources of the same type, such as duet singing voices, the prior learned by the diffusion process may not be sufficient to maintain the consistency of the source identity in the separated audio. For example, the singer may change from one to another occasionally. Tackling this problem will be useful for separating sources in a choir, or a mixture of multiple instruments with similar timbre, without acquiring large amounts of paired data. In this paper, we examine this problem in the context of duet singing voices separation, and propose a method to enforce the coherency of singer identity by splitting the mixture into overlapping segments and performing posterior sampling in an auto-regressive manner, conditioning on the previous segment. We evaluate the proposed method on the MedleyVox dataset and show that the proposed method outperforms the naive posterior sampling baseline. Our source code and the pre-trained model are publicly available at https://github.com/iamycy/duet-svs-diffusion.