# Target Speech Extraction with Conditional Diffusion Model

arXiv: https://arxiv.org/abs/2308.03987 · HTML: https://mlsanity.com/paper/2308.03987/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 8 Aug 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 24 of 24)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation](https://mlsanity.com/paper/2107.03502/) (2021)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Conditional Image Generation with Score-Based Diffusion Models](https://mlsanity.com/paper/2111.13606/) (2021)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [SpEx+: A Complete Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2005.04686/) (2020)
- [Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization](https://mlsanity.com/paper/1710.11439/) (2017)
- [Neural Target Speech Extraction: An Overview](https://mlsanity.com/paper/2301.13341/) (2023)
- [Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam](https://mlsanity.com/paper/2001.08378/) (2020)
- [Restoring degraded speech via a modified diffusion model](https://mlsanity.com/paper/2104.11347/) (2021)
- [The sound of my voice: speaker representation loss for target voice separation](https://mlsanity.com/paper/1911.02411/) (2019)

## Built on by (most cited first, 15 of 15)

- [DPM-TSE: A Diffusion Probabilistic Model for Target Sound Extraction](https://mlsanity.com/paper/2310.04567/) (2023)
- [SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer](https://mlsanity.com/paper/2409.08425/) (2024)
- [Noise-robust Speech Separation with Fast Generative Correction](https://mlsanity.com/paper/2406.07461/) (2024)
- [SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline](https://mlsanity.com/paper/2505.19314/) (2025)
- [DDTSE: Discriminative Diffusion Model for Target Speech Extraction](https://mlsanity.com/paper/2309.13874/) (2023)
- [FlowTSE: Target Speaker Extraction with Flow Matching](https://mlsanity.com/paper/2505.14465/) (2025)
- [LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Models](https://mlsanity.com/paper/2504.07402/) (2025)
- [MeanFlow-TSE: One-Step Generative Target Speaker Extraction with Mean Flow](https://mlsanity.com/paper/2512.18572/) (2025)
- [Adaptive Deterministic Flow Matching for Target Speaker Extraction](https://mlsanity.com/paper/2510.16995/) (2025)
- [Flow-TSVAD: Target-Speaker Voice Activity Detection via Latent Flow Matching](https://mlsanity.com/paper/2409.04859/) (2024)
- [Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation](https://mlsanity.com/paper/2506.08457/) (2025)
- [StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model](https://mlsanity.com/paper/2604.19635/) (2026)
- [Combining Deterministic Enhanced Conditions with Dual-Streaming Encoding for Diffusion-Based Speech Enhancement](https://mlsanity.com/paper/2505.13983/) (2025)
- [AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow](https://mlsanity.com/paper/2603.10701/) (2026)
- [Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching](https://mlsanity.com/paper/2603.12837/) (2026)

## Abstract (arXiv)

Diffusion model-based speech enhancement has received increased attention since it can generate very natural enhanced signals and generalizes well to unseen conditions. Diffusion models have been explored for several sub-tasks of speech enhancement, such as speech denoising, dereverberation, and source separation. In this paper, we investigate their use for target speech extraction (TSE), which consists of estimating the clean speech signal of a target speaker in a mixture of multi-talkers. TSE is realized by conditioning the extraction process on a clue identifying the target speaker. We show we can realize TSE using a conditional diffusion model conditioned on the clue. Besides, we introduce ensemble inference to reduce potential extraction errors caused by the diffusion process. In experiments on Libri2mix corpus, we show that the proposed diffusion model-based TSE combined with ensemble inference outperforms a comparable TSE system trained discriminatively.