# Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching

arXiv: https://arxiv.org/abs/2603.12837 · HTML: https://mlsanity.com/paper/2603.12837/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 41 of its 51 references that are in the map.
- First posted on arXiv 13 Mar 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 41)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [SpEx+: A Complete Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2005.04686/) (2020)
- [VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition](https://mlsanity.com/paper/2009.04323/) (2020)
- [DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement](https://mlsanity.com/paper/2305.08227/) (2023)
- [FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching](https://mlsanity.com/paper/2406.09286/) (2024)
- [FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching](https://mlsanity.com/paper/2505.19476/) (2025)
- [Target Speech Extraction with Conditional Diffusion Model](https://mlsanity.com/paper/2308.03987/) (2023)
- [Metis: A Foundation Speech Generation Model with Masked Generative Pre-training](https://mlsanity.com/paper/2502.03128/) (2025)
- [TSELM: Target Speaker Extraction using Discrete Tokens and Language Models](https://mlsanity.com/paper/2409.07841/) (2024)
- [Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Model](https://mlsanity.com/paper/2310.19581/) (2023)
- [Target Speaker ASR with Whisper](https://mlsanity.com/paper/2409.09543/) (2024)

## Abstract (arXiv)

Target speaker extraction (TSE) extracts the target speaker's voice from overlapping speech given a reference utterance. Existing masking-based approaches are lightweight and effective but suffer from an inability to synthesize missing content, leading to degraded perceptual quality. On the other hand, recent generative TSE models typically synthesize high-quality speech with diffusion, but require numerous iterative steps resulting in high computational costs and latency. We propose Mask2Flow-TSE, a two-stage framework combining the strengths of both paradigms. We introduce the deletion/insertion (D/I) proportion, an analytical tool that reveals early flow steps predominantly remove signal components rather than synthesize them. Based on this finding, we decouple deletion from insertion: a masking-based module handles the deletion-dominant early steps, while a single flow-matching step performs the remaining insertion for high-quality reconstruction. Specifically, the first stage uses lightweight convolution for the masking module, while the second stage employs a Diffusion Transformer (DiT) adapted for TSE with speaker conditioning. Unlike prior approaches that start from Gaussian noise, our method starts from the masked spectrogram, enabling high-quality reconstruction in a single inference step. Experiments show that Mask2Flow-TSE produces high-quality extractions with only 85M parameters and one-step inference, while preserving clean single-speaker inputs with minimal degradation.