# Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario

arXiv: https://arxiv.org/abs/2005.07272 · HTML: https://mlsanity.com/paper/2005.07272/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2020.

## Built on by (most cited first, 30 of 68)

- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [AVA-AVD: Audio-Visual Speaker Diarization in the Wild](https://mlsanity.com/paper/2111.14448/) (2021)
- [Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR](https://mlsanity.com/paper/2110.03151/) (2021)
- [DiarizationLM: Speaker Diarization Post-Processing with Large Language Models](https://mlsanity.com/paper/2401.03506/) (2024)
- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems](https://mlsanity.com/paper/2409.06656/) (2024)
- [Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition](https://mlsanity.com/paper/2204.03793/) (2022)
- [Multi-scale Speaker Diarization with Dynamic Scale Weighting](https://mlsanity.com/paper/2203.15974/) (2022)
- [A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio](https://mlsanity.com/paper/2107.02852/) (2021)
- [Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering](https://mlsanity.com/paper/2210.13690/) (2022)
- [Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition](https://mlsanity.com/paper/2408.09215/) (2024)
- [The VoxCeleb Speaker Recognition Challenge: A Retrospective](https://mlsanity.com/paper/2408.14886/) (2024)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [USTC-NELSLIP System Description for DIHARD-III Challenge](https://mlsanity.com/paper/2103.10661/) (2021)
- [End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings](https://mlsanity.com/paper/2105.02096/) (2021)
- [Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarization](https://mlsanity.com/paper/2208.13085/) (2022)
- [A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetings](https://mlsanity.com/paper/2203.16834/) (2022)
- [In search of strong embedding extractors for speaker diarisation](https://mlsanity.com/paper/2210.14682/) (2022)
- [Adapting Speaker Embeddings for Speaker Diarisation](https://mlsanity.com/paper/2104.02879/) (2021)
- [End-to-End Joint Target and Non-Target Speakers ASR](https://mlsanity.com/paper/2306.02273/) (2023)
- [Mixture Encoder for Joint Speech Separation and Recognition](https://mlsanity.com/paper/2306.12173/) (2023)
- [LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization](https://mlsanity.com/paper/2409.00819/) (2024)
- [Focus Agent: LLM-Powered Virtual Focus Group](https://mlsanity.com/paper/2409.01907/) (2024)
- [Multi-Channel End-to-End Neural Diarization with Distributed Microphones](https://mlsanity.com/paper/2110.04694/) (2021)
- [Royalflush Speaker Diarization System for ICASSP 2022 Multi-channel Multi-party Meeting Transcription Challenge](https://mlsanity.com/paper/2202.04814/) (2022)
- [The Conversational Short-phrase Speaker Diarization (CSSD) Task: Dataset, Evaluation Metric and Baselines](https://mlsanity.com/paper/2208.08042/) (2022)
- [A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers](https://mlsanity.com/paper/2304.07842/) (2023)
- [Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding with Sequence-to-Sequence Architecture](https://mlsanity.com/paper/2309.09180/) (2023)
- [Self-supervised Pretraining for Robust Personalized Voice Activity Detection in Adverse Conditions](https://mlsanity.com/paper/2312.16613/) (2023)
- [Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications](https://mlsanity.com/paper/2403.06570/) (2024)

## Abstract (arXiv)

Speaker diarization for real-life scenarios is an extremely challenging problem. Widely used clustering-based diarization approaches perform rather poorly in such conditions, mainly due to the limited ability to handle overlapping speech. We propose a novel Target-Speaker Voice Activity Detection (TS-VAD) approach, which directly predicts an activity of each speaker on each time frame. TS-VAD model takes conventional speech features (e.g., MFCC) along with i-vectors for each speaker as inputs. A set of binary classification output layers produces activities of each speaker. I-vectors can be estimated iteratively, starting with a strong clustering-based diarization. We also extend the TS-VAD approach to the multi-microphone case using a simple attention mechanism on top of hidden representations extracted from the single-channel TS-VAD model. Moreover, post-processing strategies for the predicted speaker activity probabilities are investigated. Experiments on the CHiME-6 unsegmented data show that TS-VAD achieves state-of-the-art results outperforming the baseline x-vector-based system by more than 30% Diarization Error Rate (DER) abs.