# Flow-TSVAD: Target-Speaker Voice Activity Detection via Latent Flow Matching

arXiv: https://arxiv.org/abs/2409.04859 · HTML: https://mlsanity.com/paper/2409.04859/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 29 of its 33 references that are in the map.
- First posted on arXiv 7 Sep 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks](https://mlsanity.com/paper/2012.14952/) (2020)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [BUT System Description to VoxCeleb Speaker Recognition Challenge 2019](https://mlsanity.com/paper/1910.12592/) (2019)
- [DiffusionInst: Diffusion Model for Instance Segmentation](https://mlsanity.com/paper/2212.02773/) (2022)
- [WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark](https://mlsanity.com/paper/2406.05763/) (2024)
- [Encoder-Decoder Based Attractors for End-to-End Neural Diarization](https://mlsanity.com/paper/2106.10654/) (2021)
- [From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization](https://mlsanity.com/paper/2204.00890/) (2022)
- [DOVER-Lap: A Method for Combining Overlap-aware Diarization Outputs](https://mlsanity.com/paper/2011.01997/) (2020)
- [Target Speech Extraction with Conditional Diffusion Model](https://mlsanity.com/paper/2308.03987/) (2023)
- [Simultaneous Speech Recognition and Speaker Diarization for Monaural Dialogue Recordings with Target-Speaker Acoustic Models](https://mlsanity.com/paper/1909.08103/) (2019)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Target-Speaker Voice Activity Detection via Sequence-to-Sequence Prediction](https://mlsanity.com/paper/2210.16127/) (2022)
- [Attention-based Encoder-Decoder End-to-End Neural Diarization with Embedding Enhancer](https://mlsanity.com/paper/2309.06672/) (2023)
- [Attention-based Encoder-Decoder Network for End-to-End Neural Speaker Diarization with Target Speaker Attractor](https://mlsanity.com/paper/2305.10704/) (2023)
- [USED: Universal Speaker Extraction and Diarization](https://mlsanity.com/paper/2309.10674/) (2023)
- [Multi-Input Multi-Output Target-Speaker Voice Activity Detection For Unified, Flexible, and Robust Audio-Visual Speaker Diarization](https://mlsanity.com/paper/2401.08052/) (2024)

## Built on by (most cited first, 1 of 1)

- [LibriVAD: A Scalable Open Dataset with Deep Learning Benchmarks for Voice Activity Detection](https://mlsanity.com/paper/2512.17281/) (2025)

## Abstract (arXiv)

Speaker diarization is typically considered a discriminative task, using discriminative approaches to produce fixed diarization results. In this paper, we explore the use of neural network-based generative methods for speaker diarization for the first time. We implement a Flow-Matching (FM) based generative algorithm within the sequence-to-sequence target speaker voice activity detection (Seq2Seq-TSVAD) diarization system. Our experiments reveal that applying the generative method directly to the original binary label sequence space of the TS-VAD output is ineffective. To address this issue, we propose mapping the binary label sequence into a dense latent space before applying the generative algorithm and our proposed Flow-TSVAD method outperforms the Seq2Seq-TSVAD system. Additionally, we observe that the FM algorithm converges rapidly during the inference stage, requiring only two inference steps to achieve promising results. As a generative model, Flow-TSVAD allows for sampling different diarization results by running the model multiple times. Moreover, ensembling results from various sampling instances further enhances diarization performance.