# Multi-target Extractor and Detector for Unknown-number Speaker Diarization

arXiv: https://arxiv.org/abs/2203.16007 · HTML: https://mlsanity.com/paper/2203.16007/

## Key facts

- Cited by 6 papers in the map (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [The Second DIHARD Diarization Challenge: Dataset, task, and baselines](https://mlsanity.com/paper/1906.07839/) (2019)
- [Fully Supervised Speaker Diarization](https://mlsanity.com/paper/1810.04719/) (2018)
- [Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection](https://mlsanity.com/paper/1910.11646/) (2019)
- [End-to-End Speaker Diarization as Post-Processing](https://mlsanity.com/paper/2012.10055/) (2020)
- [Speaker Diarization with Region Proposal Network](https://mlsanity.com/paper/2002.06220/) (2020)
- [End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification](https://mlsanity.com/paper/2003.02966/) (2020)
- [Speaker Diarization with Lexical Information](https://mlsanity.com/paper/2004.06756/) (2020)
- [Tight integration of neural- and clustering-based diarization through deep unfolding of infinite Gaussian mixture model](https://mlsanity.com/paper/2202.06524/) (2022)
- [DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding](https://mlsanity.com/paper/2105.13802/) (2021)
- [The Speed Submission to DIHARD II: Contributions & Lessons Learned](https://mlsanity.com/paper/1911.02388/) (2019)

## Built on by (most cited first, 6 of 6)

- [Attention-based Encoder-Decoder End-to-End Neural Diarization with Embedding Enhancer](https://mlsanity.com/paper/2309.06672/) (2023)
- [Overview of Speaker Modeling and Its Applications: From the Lens of Deep Speaker Representation Learning](https://mlsanity.com/paper/2407.15188/) (2024)
- [Attention-based Encoder-Decoder Network for End-to-End Neural Speaker Diarization with Target Speaker Attractor](https://mlsanity.com/paper/2305.10704/) (2023)
- [Sequence-to-Sequence Neural Diarization with Automatic Speaker Detection and Representation](https://mlsanity.com/paper/2411.13849/) (2024)
- [Multi-Input Multi-Output Target-Speaker Voice Activity Detection For Unified, Flexible, and Robust Audio-Visual Speaker Diarization](https://mlsanity.com/paper/2401.08052/) (2024)
- [End-to-end Online Speaker Diarization with Target Speaker Tracking](https://mlsanity.com/paper/2310.08696/) (2023)

## Abstract (arXiv)

Strong representations of target speakers can help extract important information about speakers and detect corresponding temporal regions in multi-speaker conversations. In this study, we propose a neural architecture that simultaneously extracts speaker representations consistent with the speaker diarization objective and detects the presence of each speaker on a frame-by-frame basis regardless of the number of speakers in a conversation. A speaker representation (called z-vector) extractor and a time-speaker contextualizer, implemented by a residual network and processing data in both temporal and speaker dimensions, are integrated into a unified framework. Tests on the CALLHOME corpus show that our model outperforms most of the methods proposed so far. Evaluations in a more challenging case with simultaneous speakers ranging from 2 to 7 show that our model achieves 6.4% to 30.9% relative diarization error rate reductions over several typical baselines.