# Speaker Embedding-aware Neural Diarization: an Efficient Framework for Overlapping Speech Diarization in Meeting Scenarios

arXiv: https://arxiv.org/abs/2203.09767 · HTML: https://mlsanity.com/paper/2203.09767/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks](https://mlsanity.com/paper/2012.14952/) (2020)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [CN-CELEB: a challenging Chinese speaker recognition dataset](https://mlsanity.com/paper/1911.01799/) (2019)
- [The Second DIHARD Diarization Challenge: Dataset, task, and baselines](https://mlsanity.com/paper/1906.07839/) (2019)
- [Deep-FSMN for Large Vocabulary Continuous Speech Recognition](https://mlsanity.com/paper/1803.05030/) (2018)
- [End-to-end Neural Diarization: From Transformer to Conformer](https://mlsanity.com/paper/2106.07167/) (2021)
- [Summary On The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Grand Challenge](https://mlsanity.com/paper/2202.03647/) (2022)
- [Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency](https://mlsanity.com/paper/1512.08301/) (2015)
- [Deep Feed-forward Sequential Memory Networks for Speech Synthesis](https://mlsanity.com/paper/1802.09194/) (2018)
- [Cross-Channel Attention-Based Target Speaker Voice Activity Detection: Experimental Results for M2MeT Challenge](https://mlsanity.com/paper/2202.02687/) (2022)

## Built on by (most cited first, 2 of 2)

- [Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization](https://mlsanity.com/paper/2305.12927/) (2023)
- [From Modular to End-to-End Speaker Diarization](https://mlsanity.com/paper/2407.08752/) (2024)

## Abstract (arXiv)

Overlapping speech diarization has been traditionally treated as a multi-label classification problem. In this paper, we reformulate this task as a single-label prediction problem by encoding multiple binary labels into a single label with the power set, which represents the possible combinations of target speakers. This formulation has two benefits. First, the overlaps of target speakers are explicitly modeled. Second, threshold selection is no longer needed. Through this formulation, we propose the speaker embedding-aware neural diarization (SEND) framework, where a speech encoder, a speaker encoder, two similarity scorers, and a post-processing network are jointly optimized to predict the encoded labels according to the similarities between speech features and speaker embeddings. Experimental results show that SEND has a stable learning process and can be trained on highly overlapped data without extra initialization. More importantly, our method achieves the state-of-the-art performance in real meeting scenarios with fewer model parameters and lower computational complexity.